[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461359426 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002269] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006455] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009143] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.019074] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024472] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026206] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028137] *** VALIDATE tmpfs *** [ 0.029354] *** VALIDATE proc *** [ 0.030159] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032204] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033127] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035029] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039324] debug: unmapping init [mem 0xffffffff9b659000-0xffffffff9b660fff] [ 0.041173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043027] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.045921] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047007] ... fixed-purpose events: 3 [ 0.047825] ... event mask: 000000070000000f [ 0.048257] rcu: Hierarchical SRCU implementation. [ 0.050503] smp: Bringing up secondary CPUs ... [ 0.051630] x86: Booting SMP configuration: [ 0.052024] .... node #0, CPUs: #1 #2 #3 [ 0.056833] smp: Brought up 1 node, 4 CPUs [ 0.059016] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.100107] node 0 deferred pages initialised in 36ms [ 0.102156] devtmpfs: initialized [ 0.103226] x86/mm: Memory block size: 128MB [ 0.105271] gcov: version magic: 0x41383552 [ 0.107178] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.111333] pinctrl core: initialized pinctrl subsystem [ 0.113277] [ 0.113698] ************************************************************* [ 0.116013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118011] ** ** [ 0.120012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.123012] ** ** [ 0.125011] ** This means that this kernel is built to expose internal ** [ 0.128009] ** IOMMU data structures, which may compromise security on ** [ 0.130012] ** your system. ** [ 0.132011] ** ** [ 0.134011] ** If you see this message and you are not debugging the ** [ 0.136008] ** kernel, report this immediately to your vendor! ** [ 0.138010] ** ** [ 0.139008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141009] ************************************************************* [ 0.143604] NET: Registered protocol family 16 [ 0.145337] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.147044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.149045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.151578] cpuidle: using governor menu [ 0.153014] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.155284] PCI: Using configuration type 1 for base access [ 0.157137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168172] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173101] cryptd: max_cpu_qlen set to 1000 [ 0.176258] ACPI: Added _OSI(Module Device) [ 0.177000] ACPI: Added _OSI(Processor Device) [ 0.178010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.182461] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188201] ACPI: Interpreter enabled [ 0.189000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190018] ACPI: Using IOAPIC for interrupt routing [ 0.191115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.194505] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205582] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217035] acpiphp: Slot [2] registered [ 0.218103] acpiphp: Slot [5] registered [ 0.220157] acpiphp: Slot [6] registered [ 0.221129] acpiphp: Slot [7] registered [ 0.223118] acpiphp: Slot [8] registered [ 0.225116] acpiphp: Slot [9] registered [ 0.226094] acpiphp: Slot [10] registered [ 0.227093] acpiphp: Slot [3] registered [ 0.228102] acpiphp: Slot [4] registered [ 0.229077] acpiphp: Slot [11] registered [ 0.230076] acpiphp: Slot [12] registered [ 0.231073] acpiphp: Slot [13] registered [ 0.232080] acpiphp: Slot [14] registered [ 0.233122] acpiphp: Slot [15] registered [ 0.234152] acpiphp: Slot [16] registered [ 0.235108] acpiphp: Slot [17] registered [ 0.237100] acpiphp: Slot [18] registered [ 0.238100] acpiphp: Slot [19] registered [ 0.240108] acpiphp: Slot [20] registered [ 0.241143] acpiphp: Slot [21] registered [ 0.243172] acpiphp: Slot [22] registered [ 0.244098] acpiphp: Slot [23] registered [ 0.245108] acpiphp: Slot [24] registered [ 0.247098] acpiphp: Slot [25] registered [ 0.248097] acpiphp: Slot [26] registered [ 0.249098] acpiphp: Slot [27] registered [ 0.250157] acpiphp: Slot [28] registered [ 0.252106] acpiphp: Slot [29] registered [ 0.253094] acpiphp: Slot [30] registered [ 0.255101] acpiphp: Slot [31] registered [ 0.256136] PCI host bridge to bus 0000:00 [ 0.258021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.264028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.271223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.273927] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.277179] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285601] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.290383] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297503] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.299973] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.304845] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.311015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.323022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.330016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334115] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.344018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.355016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393673] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.407016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.424016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.433078] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.445016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.452017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.468019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.477648] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.493017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.502017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.517020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.530305] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.541018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.560017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.568581] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.574020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.581017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.597013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.610096] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.613480] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.615583] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.619482] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.622309] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.627034] iommu: Default domain type: Passthrough [ 0.628447] SCSI subsystem initialized [ 0.630139] ACPI: bus type USB registered [ 0.631082] usbcore: registered new interface driver usbfs [ 0.633125] usbcore: registered new interface driver hub [ 0.635103] usbcore: registered new device driver usb [ 0.636147] pps_core: LinuxPPS API ver. 1 registered [ 0.638012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.641093] PTP clock support registered [ 0.643165] EDAC MC: Ver: 3.0.0 [ 0.645147] PCI: Using ACPI for IRQ routing [ 0.647488] NetLabel: Initializing [ 0.649012] NetLabel: domain hash size = 128 [ 0.650008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.653084] NetLabel: unlabeled traffic allowed by default [ 0.655002] vgaarb: loaded [ 0.656302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.664000] clocksource: Switched to clocksource kvm-clock [ 0.760316] VFS: Disk quotas dquot_6.6.0 [ 0.761552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.762994] *** VALIDATE ramfs *** [ 0.763656] *** VALIDATE hugetlbfs *** [ 0.764581] pnp: PnP ACPI init [ 0.766191] pnp: PnP ACPI: found 6 devices [ 0.782468] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.784528] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.785745] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.787020] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.788402] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.790769] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.793369] NET: Registered protocol family 2 [ 0.795387] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.798958] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.802293] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.807506] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.810827] TCP: Hash tables configured (established 65536 bind 65536) [ 0.812808] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.815015] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816922] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818906] NET: Registered protocol family 1 [ 0.820858] RPC: Registered named UNIX socket transport module. [ 0.822167] RPC: Registered udp transport module. [ 0.823193] RPC: Registered tcp transport module. [ 0.824185] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.825676] NET: Registered protocol family 44 [ 0.826972] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.828370] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.829926] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.831354] PCI: CLS 0 bytes, default 64 [ 0.832358] Unpacking initramfs... [ 2.212943] debug: unmapping init [mem 0xffff9f267cc54000-0xffff9f267ffbffff] [ 2.216460] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.218104] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.220282] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.702914] Initialise system trusted keyrings [ 2.703929] Key type blacklist registered [ 2.705656] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.713718] zbud: loaded [ 2.716813] *** VALIDATE nfs *** [ 2.718045] *** VALIDATE nfs4 *** [ 2.719558] pstore: using deflate compression [ 2.723856] Platform Keyring initialized [ 2.820955] NET: Registered protocol family 38 [ 2.822566] Key type asymmetric registered [ 2.824226] Asymmetric key parser 'x509' registered [ 2.825868] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.828913] io scheduler mq-deadline registered [ 2.830351] io scheduler kyber registered [ 2.832120] io scheduler bfq registered [ 2.833934] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.837048] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.840199] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.843039] ACPI: Power Button [PWRF] [ 2.848275] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.854391] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.873801] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.883392] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.904590] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.929992] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.957163] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.961184] Non-volatile memory driver v1.3 [ 2.962439] Linux agpgart interface v0.103 [ 2.991221] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.993335] vda: detected capacity change from 0 to 75108352 [ 3.011084] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.014184] vdb: detected capacity change from 0 to 1073741824 [ 3.031209] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.034468] vdc: detected capacity change from 0 to 2621440000 [ 3.057974] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.060208] vdd: detected capacity change from 0 to 2621440000 [ 3.082081] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.084515] vde: detected capacity change from 0 to 4294967296 [ 3.108536] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.110972] vdf: detected capacity change from 0 to 4294967296 [ 3.125689] libphy: Fixed MDIO Bus: probed [ 3.143120] usbcore: registered new interface driver usbserial_generic [ 3.145549] usbserial: USB Serial support registered for generic [ 3.147850] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.151883] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.153514] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.155838] mousedev: PS/2 mouse device common for all mice [ 3.157924] rtc_cmos 00:05: RTC can wake from S4 [ 3.160392] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.160639] rtc_cmos 00:05: registered as rtc0 [ 3.164599] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.167366] intel_pstate: CPU model not supported [ 3.169547] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.172889] hid: raw HID events driver (C) Jiri Kosina [ 3.175312] usbcore: registered new interface driver usbhid [ 3.177594] usbhid: USB HID core driver [ 3.179307] drop_monitor: Initializing network drop monitor service [ 3.181963] Initializing XFRM netlink socket [ 3.183897] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.184195] NET: Registered protocol family 10 [ 3.189536] Segment Routing with IPv6 [ 3.191174] NET: Registered protocol family 17 [ 3.193478] mpls_gso: MPLS GSO support [ 3.200158] RAS: Correctable Errors collector initialized. [ 3.202143] AVX version of gcm_enc/dec engaged. [ 3.203631] AES CTR mode by8 optimization enabled [ 3.274414] sched_clock: Marking stable (3274393863, 0)->(4045076302, -770682439) [ 3.277561] registered taskstats version 1 [ 3.279625] Loading compiled-in X.509 certificates [ 3.281118] zswap: loaded using pool lzo/zbud [ 3.301455] Key type big_key registered [ 3.311820] Key type encrypted registered [ 3.312776] ima: No TPM chip found, activating TPM-bypass! [ 3.313965] ima: Allocated hash algorithm: sha1 [ 3.315357] ima: No architecture policies found [ 3.317166] evm: Initialising EVM extended attributes: [ 3.319127] evm: security.selinux [ 3.320320] evm: security.ima [ 3.321386] evm: security.capability [ 3.322814] evm: HMAC attrs: 0x1 [ 3.325199] rtc_cmos 00:05: setting system clock to 2026-09-13 10:50:46 UTC (1789296646) [ 3.331197] debug: unmapping init [mem 0xffffffff9c603000-0xffffffff9c7fffff] [ 3.333836] debug: unmapping init [mem 0xffffffff9b382000-0xffffffff9b658fff] [ 3.343088] Write protecting the kernel read-only data: 28672k [ 3.345863] debug: unmapping init [mem 0xffffffff99a03000-0xffffffff99bfffff] [ 3.347607] debug: unmapping init [mem 0xffffffff9a314000-0xffffffff9a3fffff] [ 3.375730] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.381665] systemd[1]: Detected virtualization kvm. [ 3.383630] systemd[1]: Detected architecture x86-64. [ 3.385539] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.411893] systemd[1]: No hostname configured. [ 3.413735] systemd[1]: Set hostname to . [ 3.415905] random: systemd: uninitialized urandom read (16 bytes read) [ 3.418678] systemd[1]: Initializing machine ID from random generator. [ 3.455130] random: ln: uninitialized urandom read (6 bytes read) [ 3.526357] random: systemd: uninitialized urandom read (16 bytes read) [ 3.528975] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.533325] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.538254] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.201648] device-mapper: uevent: version 1.0.3 [ 4.203400] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.931309] virtio_net virtio0 ens2: renamed from eth0 [ 4.939005] random: fast init done [ 4.990094] scsi host0: ata_piix [ 5.017125] scsi host1: ata_piix [ 5.018835] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.021377] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.328653] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.832076] random: crng init done [ 9.833556] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.280379] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.533491] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.793566] SELinux: Disabled at runtime. [ 11.853427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.860828] systemd[1]: Detected virtualization kvm. [ 11.862782] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.341451] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.344434] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.348190] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.354391] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.357400] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.363757] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.368493] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ 12.441037] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.796949] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.159443] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.180936] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.328860] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.337751] EDAC sbridge: Ver: 1.1.2 [ 14.989490] Key type dns_resolver registered [ 15.285436] NFS: Registering the id_resolver key type [ 15.287454] Key type id_resolver registered [ 15.289027] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 33.989607] hrtimer: interrupt took 10881938 ns [ 71.070324] libcfs: loading out-of-tree module taints kernel. [ 71.169867] Key type ._llcrypt registered [ 71.171920] Key type .llcrypt registered [ 71.303601] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 90.450795] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 92.078564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.111400] alg: No test for adler32 (adler32-zlib) [ 93.752757] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 94.899854] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 96.729046] Key type lgssc registered [ 98.935182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.701678] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.784211] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 179.547335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.657124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.915586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.937508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.051466] Lustre: lustre-MDT0000: new disk, initializing [ 181.197927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.240141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.176203] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.479187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.673253] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.713828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.720536] Lustre: Skipped 1 previous similar message [ 201.878336] Lustre: lustre-MDT0001: new disk, initializing [ 201.982435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.026384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.035917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.573923] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.411606] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.070714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.305593] Lustre: lustre-OST0000: new disk, initializing [ 222.309393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.314522] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.376250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.952738] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.003683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.013643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.080946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.091129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.220239] Lustre: lustre-OST0001: new disk, initializing [ 243.223512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 243.228547] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.296774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.950596] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.422053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.454176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.515055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.709656] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.323676] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.550851] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.274697] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 283.735894] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 286.322174] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 288.691018] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 290.036592] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 06:55:32 EDT 2026 [ 310.353870] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 312.101478] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.208773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.217712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.240791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.260261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.261185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.282322] Lustre: Skipped 2 previous similar messages [ 332.286034] Lustre: Skipped 2 previous similar messages [ 334.347666] Lustre: server umount lustre-MDT0000 complete [ 338.186907] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789296981 with bad export cookie 15237974581556428997 [ 338.197033] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.495929] LustreError: 9543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.497359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.506788] LustreError: 9543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 342.510923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.510931] Lustre: Skipped 1 previous similar message [ 342.553353] Lustre: Skipped 1 previous similar message [ 343.530664] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.536148] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 344.674165] Lustre: server umount lustre-MDT0001 complete [ 348.776982] Lustre: server umount lustre-OST0000 complete [ 353.529586] Lustre: server umount lustre-OST0001 complete [ 362.811355] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 373.966607] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 413.873845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.135290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.161735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.247909] Lustre: lustre-MDT0000: new disk, initializing [ 414.344735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.376536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.677933] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 429.903822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.101173] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.213468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.217815] Lustre: Skipped 1 previous similar message [ 430.323496] Lustre: lustre-MDT0001: new disk, initializing [ 430.496425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.568749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.584571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.993755] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 447.110662] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.484787] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.659057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.907730] Lustre: lustre-OST0000: new disk, initializing [ 460.912164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.919832] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.043850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.664752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.682348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.809523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.739260] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 479.037801] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 484.647451] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.006550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.968716] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.217811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.664816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.669802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.686380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.692190] Lustre: Skipped 1 previous similar message [ 498.709639] Lustre: Skipped 4 previous similar messages [ 499.892622] Lustre: server umount lustre-OST0000 complete [ 507.364049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.372770] LustreError: Skipped 1 previous similar message [ 507.382133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.401286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.679197] Lustre: server umount lustre-MDT0000 complete [ 514.017081] LustreError: 15967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.041061] LustreError: 15967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 514.967063] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297158 with bad export cookie 15237974581556430530 [ 514.970626] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.978393] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 515.292919] Lustre: server umount lustre-MDT0001 complete [ 519.448688] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:59:21 (1789297161) [ 521.958995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 523.864915] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:59:25 (1789297165) [ 531.998771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.494043] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.558710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.224796] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 537.575519] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.534802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.283541] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 562.245580] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 569.124858] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.064481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 582.972507] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 591.386897] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.104481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.108327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.115428] LustreError: Skipped 1 previous similar message [ 603.116849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.130453] Lustre: Skipped 2 previous similar messages [ 603.154646] Lustre: Skipped 3 previous similar messages [ 609.269700] Lustre: server umount lustre-OST0000 complete [ 620.406277] Lustre: server umount lustre-MDT0000 complete [ 623.591938] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.626742] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 624.857177] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297268 with bad export cookie 15237974581556431412 [ 624.871209] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.871989] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 625.257789] Lustre: server umount lustre-MDT0001 complete [ 638.120416] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 641.357626] Key type lgssc unregistered [ 641.644408] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.659719] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.672991] LNet: Removed LNI 192.168.202.138@tcp [ 643.044170] Key type .llcrypt unregistered [ 643.046676] Key type ._llcrypt unregistered [ 665.497750] Key type ._llcrypt registered [ 665.499646] Key type .llcrypt registered [ 665.615655] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 666.919381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.214653] alg: No test for adler32 (adler32-zlib) [ 668.296705] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 668.583381] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 670.271656] Key type lgssc registered [ 671.569561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.945674] loop: module loaded [ 679.059394] loop0: detected capacity change from 0 to 204800000 [ 679.308512] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.327853] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.729441] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.057728] loop0: detected capacity change from 0 to 204800000 [ 686.246794] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.259852] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.305544] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.142396] loop0: detected capacity change from 0 to 204800000 [ 697.038170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.091459] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.359173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 698.446978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 698.514277] Lustre: lustre-MDT0000: new disk, initializing [ 698.601800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.625336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 703.159333] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 712.994720] loop1: detected capacity change from 0 to 204800000 [ 717.725457] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.098641] Lustre: lustre-OST07c6: new disk, initializing [ 718.104154] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 718.108692] Lustre: Skipped 1 previous similar message [ 718.116962] Lustre: 26203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 718.242363] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 724.567042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 724.574599] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 724.626858] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 725.094381] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 730.233773] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 730.251885] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 730.268807] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 730.281957] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 730.291134] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 730.298404] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 751.238613] Lustre: 26720:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 751.251850] Lustre: 26720:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.257602] Lustre: 26720:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.265815] Lustre: 26720:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 751.272416] Lustre: 26720:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 751.277461] Lustre: 26720:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.022769] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 756.053911] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 756.063641] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 756.077843] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.098185] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 756.112809] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.127782] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 756.144430] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.165595] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.175178] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.187240] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.198470] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.430761] Lustre: 26720:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 759.451979] Lustre: 26720:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 759.463571] Lustre: 26720:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 759.483452] Lustre: 26720:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.495413] Lustre: 26720:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.507395] Lustre: 26720:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.519372] Lustre: 26720:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 759.531211] Lustre: 26720:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.539708] Lustre: 26720:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.552062] Lustre: 26720:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.566827] Lustre: 26720:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.575661] Lustre: 26720:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.422498] Lustre: 26720:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 764.432114] Lustre: 26720:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 764.439780] Lustre: 26720:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.452415] Lustre: 26720:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.461471] Lustre: 26720:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.469412] Lustre: 26720:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.479642] Lustre: 26720:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 764.489811] Lustre: 26720:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.499763] Lustre: 26720:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.507467] Lustre: 26720:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.527981] Lustre: 26720:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.534720] Lustre: 26720:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.077911] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 776.087741] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 776.097942] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.105271] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.114634] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.122852] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.132513] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 776.138576] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.146903] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.152370] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.158464] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.170683] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 826.863321] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.888694] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.913416] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.577185] Lustre: server umount lustre-OST07c6 complete [ 840.074621] Lustre: server umount lustre-MDT0000 complete [ 857.048897] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 860.127175] Key type lgssc unregistered [ 860.526748] LNet: 28762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.532460] LNetError: 28762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.548061] LNet: Removed LNI 192.168.202.138@tcp [ 862.193389] Key type .llcrypt unregistered [ 862.196361] Key type ._llcrypt unregistered [ 885.890272] Key type ._llcrypt registered [ 885.894368] Key type .llcrypt registered [ 886.033555] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 887.158104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.286619] alg: No test for adler32 (adler32-zlib) [ 888.530739] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 888.835402] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 890.527198] Key type lgssc registered [ 891.884937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 900.163113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 900.182164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.799244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.296220] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 911.363770] LustreError: 30031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.388514] LustreError: 30031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 914.971955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.324196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 920.520514] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 928.088333] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 933.766222] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 940.398450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.910701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.058080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 947.548435] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 955.608144] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 975.845541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 975.854511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.856243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.879747] Lustre: Skipped 1 previous similar message [ 977.138521] Lustre: server umount lustre-OST0000 complete [ 981.984785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 981.987432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.994730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.997378] Lustre: Skipped 1 previous similar message [ 986.083726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.086235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.137074] Lustre: Skipped 1 previous similar message [ 987.470547] Lustre: server umount lustre-MDT0000 complete [ 991.200329] LustreError: 30030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.215106] LustreError: 30030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 991.648652] LustreError: 30011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297634 with bad export cookie 17155745614829046394 [ 991.655340] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.660377] LustreError: 30011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 992.114209] Lustre: server umount lustre-MDT0001 complete [ 1004.154041] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1007.340358] Key type lgssc unregistered [ 1007.728735] LNet: 33764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.737422] LNetError: 33764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.758971] LNet: Removed LNI 192.168.202.138@tcp [ 1008.938773] Key type .llcrypt unregistered [ 1008.941667] Key type ._llcrypt unregistered [ 1028.781547] Key type ._llcrypt registered [ 1028.783752] Key type .llcrypt registered [ 1028.904965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1030.143222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.360197] alg: No test for adler32 (adler32-zlib) [ 1031.331722] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1031.579769] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1033.287191] Key type lgssc registered [ 1034.461568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1043.788062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1043.815636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.462495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.948668] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1055.715631] LustreError: 35037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.737801] LustreError: 35037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1059.775365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.353107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1066.168904] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1074.200355] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.806444] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.378724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.776314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.800576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1095.912206] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1104.912235] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.659794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.664439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.676976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.736697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.745549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.757508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.800302] Lustre: server umount lustre-OST0000 complete [ 1126.881325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.895638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.911649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.919883] Lustre: Skipped 1 previous similar message [ 1130.982698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.984526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1131.038810] Lustre: Skipped 1 previous similar message [ 1136.097245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.103169] Lustre: Skipped 1 previous similar message [ 1141.217964] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1141.419639] Lustre: server umount lustre-MDT0000 complete [ 1145.103774] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297788 with bad export cookie 3984734206251312492 [ 1145.107992] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.127156] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1145.475546] Lustre: server umount lustre-MDT0001 complete [ 1157.210406] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1159.984181] Key type lgssc unregistered [ 1160.307648] LNet: 38570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.316232] LNetError: 38570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.331467] LNet: Removed LNI 192.168.202.138@tcp [ 1161.237690] Key type .llcrypt unregistered [ 1161.242177] Key type ._llcrypt unregistered [ 1188.428853] Key type ._llcrypt registered [ 1188.432355] Key type .llcrypt registered [ 1188.607718] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1190.416260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.496547] alg: No test for adler32 (adler32-zlib) [ 1191.592866] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1191.865171] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1193.567280] Key type lgssc registered [ 1194.957544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1203.416807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1203.439262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.973251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.434467] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1215.456748] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.475485] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1218.930169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.511738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1225.354534] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1234.585466] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1240.858233] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1248.054973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.526961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.644652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1255.420483] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1263.502458] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1270.762327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1270.769836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.778248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.370407] Lustre: server umount lustre-MDT0000 complete [ 1274.339611] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.340947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.367396] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1274.413915] Lustre: Skipped 1 previous similar message [ 1276.316944] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.336846] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1276.979250] LustreError: 39966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297920 with bad export cookie 6273383895806201667 [ 1276.993958] LustreError: 39966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1276.996651] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1279.458560] LustreError: 39982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.459444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.545564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1279.562316] Lustre: Skipped 1 previous similar message [ 1281.446849] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 1283.544151] Lustre: server umount lustre-MDT0001 complete [ 1305.454131] Lustre: server umount lustre-OST0000 complete [ 1323.197663] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1326.052164] Key type lgssc unregistered [ 1326.294045] LNet: 43843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.300287] LNetError: 43843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.344694] LNet: Removed LNI 192.168.202.138@tcp [ 1328.376929] Key type .llcrypt unregistered [ 1328.379514] Key type ._llcrypt unregistered [ 1353.093311] Key type ._llcrypt registered [ 1353.094903] Key type .llcrypt registered [ 1353.243255] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1354.582709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.768846] alg: No test for adler32 (adler32-zlib) [ 1355.840469] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1356.034329] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1357.679175] Key type lgssc registered [ 1358.857978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.039433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.068254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.504738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.100274] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1378.786373] LustreError: 45271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.808590] LustreError: 45271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1381.514837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.820757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1386.369990] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1394.547921] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1400.599188] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1407.964376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.303498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.433656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1415.656875] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1423.688739] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1434.080719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.091253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.124532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.621118] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 1437.168860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.171421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.988804] Lustre: server umount lustre-OST0000 complete [ 1457.288391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.290628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1457.310280] Lustre: Skipped 1 previous similar message [ 1457.325943] Lustre: Skipped 3 previous similar messages [ 1463.713742] Lustre: server umount lustre-MDT0000 complete [ 1463.787991] LustreError: 46279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.805163] LustreError: 46279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1467.962573] LustreError: 45251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298111 with bad export cookie 16643679333348113872 [ 1467.964758] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.979222] LustreError: 45251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.293626] Lustre: server umount lustre-MDT0001 complete [ 1485.126339] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1488.049905] Key type lgssc unregistered [ 1488.390703] LNet: 48987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.395761] LNetError: 48987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.408576] LNet: Removed LNI 192.168.202.138@tcp [ 1489.520913] Key type .llcrypt unregistered [ 1489.523120] Key type ._llcrypt unregistered [ 1513.820576] Key type ._llcrypt registered [ 1513.823759] Key type .llcrypt registered [ 1513.968090] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1515.298850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.359043] alg: No test for adler32 (adler32-zlib) [ 1516.510094] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1516.765910] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1518.463864] Key type lgssc registered [ 1519.939626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1527.873264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1527.885775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.406553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.808720] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1539.554749] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.584572] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1543.744641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.359278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.598058] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1558.538024] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1564.794296] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1571.879217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.259602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.345967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1578.805677] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1586.236555] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1596.144488] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1599.472945] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1611.840531] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-13 7h17m33s [ 1613.552846] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1615.840595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1615.852089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.867987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.929964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.930812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.945996] Lustre: Skipped 1 previous similar message [ 1616.968617] Lustre: Skipped 1 previous similar message [ 1618.032785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.046527] Lustre: Skipped 2 previous similar messages [ 1620.069051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.081092] Lustre: Skipped 3 previous similar messages [ 1622.070618] Lustre: server umount lustre-MDT0000 complete [ 1622.139653] LustreError: 50397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.164909] LustreError: 50397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1623.200974] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.316844] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.333763] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1625.676466] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1629.414485] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.462345] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1633.438117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.704130] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.118465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.186290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1635.124367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1638.911385] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1642.879720] Lustre: DEBUG MARKER: Wait for df (36633) ... [ 1644.540787] Lustre: DEBUG MARKER: done [ 1649.633661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.643189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.658795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.664664] Lustre: Skipped 2 previous similar messages [ 1654.367326] Lustre: server umount lustre-OST0000 complete [ 1660.896785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1660.899353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.913803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.913811] Lustre: Skipped 2 previous similar messages [ 1660.957512] Lustre: Skipped 2 previous similar messages [ 1672.159681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1672.360447] Lustre: server umount lustre-MDT0000 complete [ 1676.203261] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.229897] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 1676.417938] LustreError: 50383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298319 with bad export cookie 13087632402534474919 [ 1676.434459] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.444943] LustreError: 50383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1676.724683] Lustre: server umount lustre-MDT0001 complete [ 1689.859589] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1693.180168] Key type lgssc unregistered [ 1693.478473] LNet: 55329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.502373] LNetError: 55329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.579393] LNet: Removed LNI 192.168.202.138@tcp [ 1695.616162] Key type .llcrypt unregistered [ 1695.622910] Key type ._llcrypt unregistered [ 1739.939111] Key type ._llcrypt registered [ 1739.942931] Key type .llcrypt registered [ 1740.102244] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1741.840346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.900765] alg: No test for adler32 (adler32-zlib) [ 1743.317666] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1743.721212] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1745.520115] Key type lgssc registered [ 1746.836387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1756.081420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1756.108593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.480855] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1757.555341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1757.864901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.968226] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1776.277144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.450057] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1776.499049] Lustre: 57534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1776.851866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.602778] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1790.700077] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1797.259309] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1804.374141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.514176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1804.721970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.284420] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1814.133911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1819.219957] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.385309] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-13 7h21m07s [ 1826.922196] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1835.695883] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1838.530460] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1838.606060] LustreError: 59402:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1859.032242] Lustre: lustre-MDT0000: Client 563c2c13-cbc0-43c7-8394-f37a57b42c4d (at 192.168.202.38@tcp) reconnecting [ 1878.655108] LustreError: 59402:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1881.418603] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1884.097838] Lustre: DEBUG MARKER: done [ 1890.792158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.803616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.820641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.303093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.315599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.332549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.342527] Lustre: Skipped 1 previous similar message [ 1896.517296] Lustre: server umount lustre-OST0000 complete [ 1901.536770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.567452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.761971] Lustre: server umount lustre-MDT0000 complete [ 1907.687740] LustreError: 57546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.709676] LustreError: 57546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1910.702185] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298553 with bad export cookie 17062888790271665681 [ 1910.715792] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.723378] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.053894] Lustre: server umount lustre-MDT0001 complete [ 1923.762984] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1926.944969] Key type lgssc unregistered [ 1927.343611] LNet: 61861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.357102] LNetError: 61861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.369382] LNet: Removed LNI 192.168.202.138@tcp [ 1928.316763] Key type .llcrypt unregistered [ 1928.318903] Key type ._llcrypt unregistered [ 1975.618716] Key type ._llcrypt registered [ 1975.621276] Key type .llcrypt registered [ 1975.785323] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1977.594140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.665102] alg: No test for adler32 (adler32-zlib) [ 1978.720327] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 1978.969626] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1980.695930] Key type lgssc registered [ 1981.887985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.290831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1985.331747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.598455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1986.618668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1986.759922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.717648] Lustre: Failing over lustre-MDT0000 [ 1988.102829] Lustre: server umount lustre-MDT0000 complete [ 1994.379524] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:23:56 (1789298636) [ 2001.282156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.927507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.441074] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2016.172308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.261581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2016.302830] Lustre: 64657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2016.673512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.923672] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2029.621600] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.339519] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.447895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.633350] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2042.818706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.359423] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2053.273357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2055.042947] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2060.189687] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2065.929816] Lustre: server umount lustre-MDT0000 complete [ 2067.936806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.952687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.962421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.553104] LustreError: 64650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298712 with bad export cookie 9227108526797972510 [ 2069.569427] LustreError: 64650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2069.571563] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.891254] Lustre: server umount lustre-MDT0001 complete [ 2073.729773] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2077.696301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2086.141203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.463216] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298717/real 1789298717] req@00000000a2f2939e x1876215591899520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.499258] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.522601] Lustre: Skipped 1 previous similar message [ 2095.209325] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.217441] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2095.321461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.450670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2096.287111] Lustre: 63581:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298722/real 1789298722] req@000000003ae29b83 x1876215591899648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.384110] Lustre: 63579:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298727/real 1789298727] req@0000000008ccf832 x1876215591899904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.449550] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.464644] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2101.066993] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2105.442959] Lustre: 63578:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298732/real 1789298732] req@00000000c5618ec4 x1876215591900160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.825987] LustreError: 68426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.849487] LustreError: 68426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2108.282264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.689426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.753968] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2112.828524] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2119.456713] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.091951] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2129.377075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.386196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.398495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.409406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.416097] Lustre: Skipped 2 previous similar messages [ 2135.511925] Lustre: server umount lustre-MDT0000 complete [ 2135.527285] LustreError: 68980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.545933] LustreError: 68980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2138.767538] LustreError: 64650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298781 with bad export cookie 9227108526797974785 [ 2138.773400] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.776327] LustreError: 64650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2138.998549] Lustre: server umount lustre-MDT0001 complete [ 2145.242872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.011282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.831489] Lustre: 63581:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298783/real 1789298783] req@00000000af5b2b1f x1876215591925504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.860479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.873585] Lustre: Skipped 2 previous similar messages [ 2163.049485] LustreError: 63577:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000670e1978 x1876215591926528/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.481617] LustreError: 69916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.504410] LustreError: 69916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2163.582605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.671381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2168.167845] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2170.719384] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789298797/real 1789298797] req@000000002f7f4d49 x1876215591926144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789298813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.746711] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2172.903619] LustreError: 71108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.915721] LustreError: 71108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2175.543272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.010296] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2181.109407] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2181.117591] Lustre: Skipped 1 previous similar message [ 2185.263674] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2186.405844] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.114799] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2197.707211] Lustre: server umount lustre-MDT0000 complete [ 2200.552148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.562433] LustreError: 72150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.577846] LustreError: 72150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2201.069340] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298844 with bad export cookie 9227108526797976164 [ 2201.086615] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.322131] Lustre: server umount lustre-MDT0001 complete [ 2206.479977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2209.781829] Lustre: DEBUG MARKER: files compared the same [ 2222.566841] Lustre: server umount lustre-OST0000 complete [ 2237.054255] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2240.342946] Key type lgssc unregistered [ 2240.657862] LNet: 74381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.663596] LNetError: 74381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.679617] LNet: Removed LNI 192.168.202.138@tcp [ 2241.869354] Key type .llcrypt unregistered [ 2241.874396] Key type ._llcrypt unregistered [ 2266.875092] Key type ._llcrypt registered [ 2266.884162] Key type .llcrypt registered [ 2267.090334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2268.947625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2269.114837] alg: No test for adler32 (adler32-zlib) [ 2270.246445] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2270.498155] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2272.135099] Key type lgssc registered [ 2273.866678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.488853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2282.503571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.104935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.740901] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2294.241786] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.261563] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2298.933866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.401664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.978523] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2313.350560] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2319.857840] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2326.913981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2327.147036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.209723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2334.107712] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2340.663722] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2355.168763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2355.173102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.178373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2355.183407] Lustre: Skipped 1 previous similar message [ 2355.194382] Lustre: Skipped 1 previous similar message [ 2359.653056] Lustre: server umount lustre-OST0000 complete [ 2365.411390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.428152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.440219] Lustre: Skipped 1 previous similar message [ 2369.414858] Lustre: server umount lustre-MDT0000 complete [ 2370.533401] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.541714] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2373.200104] LustreError: 76532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299016 with bad export cookie 13791115337649715190 [ 2373.204125] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.210734] LustreError: 76532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2373.535568] Lustre: server umount lustre-MDT0001 complete [ 2384.143342] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2386.534722] Key type lgssc unregistered [ 2386.953531] LNet: 79352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.962336] LNetError: 79352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2386.996594] LNet: Removed LNI 192.168.202.138@tcp [ 2387.948786] Key type .llcrypt unregistered [ 2387.955428] Key type ._llcrypt unregistered [ 2415.455906] Key type ._llcrypt registered [ 2415.458738] Key type .llcrypt registered [ 2415.524864] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2416.655993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2416.683167] alg: No test for adler32 (adler32-zlib) [ 2417.767596] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2417.950731] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2419.647950] Key type lgssc registered [ 2420.977339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2430.174173] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.214358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.791554] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2450.912169] LustreError: 80740:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 2477.663211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.101922] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2492.876087] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2500.680833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.824473] Lustre: *** cfs_fail_loc=706, val=0*** [ 2501.208539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.279312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2505.808670] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2506.208021] LustreError: 80766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.221706] LustreError: 80766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2511.330512] LustreError: 80765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.364670] LustreError: 81826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.393702] LustreError: 81826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2514.062911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.575110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.968630] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2525.823064] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2532.899803] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2536.935172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.959310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.990888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.003186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.015785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.052185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.634290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.650061] Lustre: Skipped 2 previous similar messages [ 2550.239216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2550.453575] Lustre: server umount lustre-OST0000 complete [ 2555.361324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2555.392720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.414945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2560.999670] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.018156] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2561.115806] Lustre: server umount lustre-MDT0000 complete [ 2564.884783] LustreError: 80748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299208 with bad export cookie 17492331366130540551 [ 2564.894877] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.896112] LustreError: 80748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2565.211918] Lustre: server umount lustre-MDT0001 complete [ 2576.938480] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2579.815624] Key type lgssc unregistered [ 2580.161543] LNet: 84353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.171525] LNetError: 84353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.194655] LNet: Removed LNI 192.168.202.138@tcp [ 2581.062819] Key type .llcrypt unregistered [ 2581.064605] Key type ._llcrypt unregistered [ 2608.911437] Key type ._llcrypt registered [ 2608.913911] Key type .llcrypt registered [ 2609.012667] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2610.342539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.573085] alg: No test for adler32 (adler32-zlib) [ 2611.718729] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2611.927472] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2613.616289] Key type lgssc registered [ 2615.122967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2623.536833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2623.555477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.327605] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2637.584562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.165875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.250672] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2643.437284] LustreError: 86455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.453735] LustreError: 86455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2648.544874] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.564853] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2650.685885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.987907] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.001750] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2651.014733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.244674] LustreError: 87203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.268232] LustreError: 87203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2658.571276] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2661.351497] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.393914] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2668.689480] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.724172] Lustre: MGS: Client f01110b5-687f-4ca4-8bef-4ca12717ee01 (at 0@lo) reconnecting [ 2668.754128] LustreError: 86469:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2669.140735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.247455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2674.755272] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2691.558704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.575264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.606072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.624303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.641954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.642965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.657216] Lustre: Skipped 1 previous similar message [ 2696.705945] Lustre: server umount lustre-OST0000 complete [ 2704.864872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2704.866753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.888954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.901782] Lustre: Skipped 1 previous similar message [ 2704.923470] Lustre: Skipped 1 previous similar message [ 2707.395946] Lustre: server umount lustre-MDT0000 complete [ 2709.995938] LustreError: 88378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.035752] LustreError: 88378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2711.917887] Lustre: server umount lustre-MDT0001 complete [ 2716.618661] Lustre: server umount MGS complete [ 2728.614564] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:36:10 (1789299370) [ 2763.408885] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2775.007609] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2809.565424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.412479] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2824.309505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.490450] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2824.518460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2824.593574] Lustre: lustre-MDT0001: new disk, initializing [ 2824.718130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.747075] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2824.757556] Lustre: 93647:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2824.778111] LustreError: 93647:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2829.139172] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2840.101171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.325926] Lustre: lustre-OST0000: new disk, initializing [ 2840.330346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2840.339246] Lustre: 94412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2840.416684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.600707] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2848.800261] Lustre: 94836:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000693dbd42 x1876216255653888/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789299507 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2855.475696] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2862.497080] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.525745] Lustre: MGS: Client 4a1b682c-4564-4476-a307-0deb0833b193 (at 0@lo) reconnecting [ 2862.618292] Lustre: 92915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.641854] LustreError: 93633:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2862.672704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2862.762628] Lustre: lustre-MDT0000: new disk, initializing [ 2862.840710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.866317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2863.003603] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.914761] Lustre: 95480:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f1372b03 x1876216255660544/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789299523 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2867.693721] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2869.233030] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2873.010168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2873.024702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2873.148386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2898.411298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.417545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.431553] Lustre: Skipped 1 previous similar message [ 2898.442178] Lustre: Skipped 1 previous similar message [ 2903.541712] Lustre: server umount lustre-OST0000 complete [ 2908.653930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.660934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.689518] Lustre: Skipped 1 previous similar message [ 2908.689797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.730640] Lustre: Skipped 1 previous similar message [ 2913.388796] Lustre: server umount lustre-MDT0000 complete [ 2913.763574] LustreError: 93617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.777746] LustreError: 93617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2923.446360] Lustre: server umount lustre-MDT0001 complete [ 2927.401801] Lustre: server umount MGS complete [ 2939.655857] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:39:41 (1789299581) [ 2957.805430] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2960.656113] Key type lgssc unregistered [ 2961.093053] LNet: 97978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2961.108341] LNetError: 97978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2962.149045] LNet: Removed LNI 192.168.202.138@tcp [ 2963.262301] Key type .llcrypt unregistered [ 2963.270770] Key type ._llcrypt unregistered [ 2985.987879] Key type ._llcrypt registered [ 2985.993356] Key type .llcrypt registered [ 2986.166504] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2987.464163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2987.516046] alg: No test for adler32 (adler32-zlib) [ 2988.985954] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 2989.372626] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2991.103841] Key type lgssc registered [ 2992.836772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3002.138566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3002.159696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.656402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.931395] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3013.107574] LustreError: 99203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.134490] LustreError: 99203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3017.235723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.540219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3022.142343] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3030.639240] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3036.307727] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3042.967952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.269955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.367074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3051.057074] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3060.465939] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3072.997409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.999533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.018674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3073.059923] Lustre: Skipped 1 previous similar message [ 3077.790948] Lustre: server umount lustre-OST0000 complete [ 3084.258702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3084.279408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.341254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.365346] Lustre: Skipped 1 previous similar message [ 3088.356364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.363099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.903441] Lustre: server umount lustre-MDT0000 complete [ 3093.473804] LustreError: 99199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.548037] LustreError: 99199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3094.504821] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.533995] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3095.187090] LustreError: 99183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299738 with bad export cookie 3044895476352179551 [ 3095.205419] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.209202] LustreError: 99183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3095.937503] Lustre: server umount lustre-MDT0001 complete [ 3108.554714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3111.188845] Key type lgssc unregistered [ 3111.491673] LNet: 102736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3111.499436] LNetError: 102736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3112.551619] LNet: Removed LNI 192.168.202.138@tcp [ 3113.570700] Key type .llcrypt unregistered [ 3113.576764] Key type ._llcrypt unregistered [ 3131.410261] Key type ._llcrypt registered [ 3131.413662] Key type .llcrypt registered [ 3131.583508] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3132.862453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3132.975124] alg: No test for adler32 (adler32-zlib) [ 3134.149360] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3134.509033] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3136.343188] Key type lgssc registered [ 3137.893603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3144.919440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3144.924592] LustreError: 103810:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3144.955413] LustreError: 103817:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3144.958450] LustreError: 103810:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3144.960994] LustreError: 103810:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3144.968382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.972730] LustreError: 103810:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3146.552480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.829663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.195164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3160.378719] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3166.612744] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lsmod [ 3169.583437] LustreError: 105024:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3169.586158] LustreError: 105021:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3169.609111] LustreError: 105024:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3169.628429] LustreError: 105024:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3169.646835] LustreError: 105024:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3169.705419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3170.092599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.121603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3175.906628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3175.913410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.934657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3176.436314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.460179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3176.469063] Lustre: Skipped 2 previous similar messages [ 3178.968891] Lustre: server umount lustre-MDT0000 complete [ 3180.585092] LustreError: 103855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.611422] LustreError: 103855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3182.964781] LustreError: 103830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299826 with bad export cookie 9829597502343941693 [ 3182.977329] LustreError: 103830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3182.982024] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.215548] Lustre: server umount lustre-MDT0001 complete [ 3197.528477] Lustre: server umount lustre-OST0000 complete [ 3207.123959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.047807] LustreError: 106088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.252098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.281661] LustreError: 106089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.823559] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3218.401941] LustreError: 106088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.229851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.650032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3227.140843] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3236.136437] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.788143] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3251.445197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.917598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.996521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3258.242833] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3265.723408] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3273.699221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.711370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.726655] Lustre: Skipped 1 previous similar message [ 3273.736708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.795141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.803985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3277.815847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.194052] Lustre: server umount lustre-OST0000 complete [ 3282.918597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.947891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3282.954980] Lustre: Skipped 2 previous similar messages [ 3293.168462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.174067] Lustre: Skipped 3 previous similar messages [ 3297.247150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3297.739475] Lustre: server umount lustre-MDT0000 complete [ 3298.274516] LustreError: 106089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.298870] LustreError: 106089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3301.740503] LustreError: 106070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299944 with bad export cookie 9829597502343942624 [ 3301.763824] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.241334] Lustre: server umount lustre-MDT0001 complete [ 3314.310416] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3317.593882] Key type lgssc unregistered [ 3317.952865] LNet: 109623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.966287] LNetError: 109623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3319.017495] LNet: Removed LNI 192.168.202.138@tcp [ 3320.153383] Key type .llcrypt unregistered [ 3320.157393] Key type ._llcrypt unregistered [ 3346.141737] Key type ._llcrypt registered [ 3346.148433] Key type .llcrypt registered [ 3346.255776] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3347.337067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.515493] alg: No test for adler32 (adler32-zlib) [ 3348.602548] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3348.870220] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3350.562042] Key type lgssc registered [ 3352.057809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.340218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.366308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.890955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.451732] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3372.514799] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.528138] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3375.673660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.991027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.645807] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3389.277488] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3395.955971] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3403.143874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.619267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.783446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3410.855705] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3418.985485] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.807772] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3428.268856] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3433.439975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3433.443944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.452529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.516335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.518052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.533403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.575873] Lustre: server umount lustre-OST0000 complete [ 3442.657189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.662098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.668794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.672871] Lustre: Skipped 1 previous similar message [ 3446.758240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.766316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3446.806307] Lustre: Skipped 1 previous similar message [ 3448.708600] Lustre: server umount lustre-MDT0000 complete [ 3451.876206] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.910030] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3453.349564] LustreError: 111025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300096 with bad export cookie 16806692902058825644 [ 3453.363552] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3453.381366] LustreError: 111025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3453.860223] Lustre: server umount lustre-MDT0001 complete [ 3466.955568] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3469.666568] Key type lgssc unregistered [ 3469.944493] LNet: 114681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.949606] LNetError: 114681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3469.968385] LNet: Removed LNI 192.168.202.138@tcp [ 3470.783583] Key type .llcrypt unregistered [ 3470.785123] Key type ._llcrypt unregistered [ 3489.753408] Key type ._llcrypt registered [ 3489.755699] Key type .llcrypt registered [ 3489.874439] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3491.171079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.237212] alg: No test for adler32 (adler32-zlib) [ 3492.246659] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3492.527833] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3494.239285] Key type lgssc registered [ 3495.917732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3503.375111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3503.407567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.788931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.809370] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3514.348410] LustreError: 115898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.383292] LustreError: 115898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3516.303091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.698827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3521.954713] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3528.410674] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3533.030776] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3539.810382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3540.280830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.748764] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3549.500912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3553.959331] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3561.952787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.957314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.965895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3565.046613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3565.048510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3565.053269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.113483] Lustre: Skipped 1 previous similar message [ 3567.978884] Lustre: server umount lustre-OST0000 complete [ 3573.219903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3573.225337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.230235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.269459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.277554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.278329] Lustre: server umount lustre-MDT0000 complete [ 3580.387107] LustreError: 116884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.268766] LustreError: 115879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300225 with bad export cookie 1130473017757484244 [ 3582.281546] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3582.294837] LustreError: 115879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3582.637218] Lustre: server umount lustre-MDT0001 complete [ 3594.320353] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3596.951167] Key type lgssc unregistered [ 3597.238602] LNet: 119429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3597.251869] LNetError: 119429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3597.266557] LNet: Removed LNI 192.168.202.138@tcp [ 3598.136886] Key type .llcrypt unregistered [ 3598.139656] Key type ._llcrypt unregistered [ 3620.841644] Key type ._llcrypt registered [ 3620.843931] Key type .llcrypt registered [ 3620.948875] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3621.975911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3622.075664] alg: No test for adler32 (adler32-zlib) [ 3623.162935] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3623.408735] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3625.143201] Key type lgssc registered [ 3626.379707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3634.398140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3634.429638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.482120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.973591] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3645.926846] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.961587] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3651.042205] LustreError: 120843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.004929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.437803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3656.842081] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3664.346112] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3670.402914] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3677.923239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3678.425029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.555277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3685.719230] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3695.329449] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3701.890745] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3706.011013] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3706.018195] Lustre: Skipped 2 previous similar messages [ 3711.432796] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3711.438410] Lustre: Skipped 2 previous similar messages [ 3711.611920] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3711.641087] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3724.289905] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3724.295360] Lustre: Skipped 2 previous similar messages [ 3730.523207] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp [ 3733.617476] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.38/24@tcp in log lustre-MDT0000 [ 3733.626413] Lustre: Skipped 2 previous similar messages [ 3740.763944] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 3740.778422] Lustre: Skipped 1 previous similar message [ 3747.817909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3747.830235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.851354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.147504] Lustre: server umount lustre-OST0000 complete [ 3753.447211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.450388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.484211] Lustre: Skipped 1 previous similar message [ 3758.562371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.573820] Lustre: Skipped 2 previous similar messages [ 3763.681556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.690447] Lustre: Skipped 1 previous similar message [ 3764.639246] Lustre: 120154:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789300391/real 1789300391] req@00000000271080ed x1876217315781376/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789300407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3764.673879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.239179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3766.476400] Lustre: server umount lustre-MDT0000 complete [ 3769.829272] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.831503] Lustre: 120154:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789300396/real 1789300396] req@000000001c8a9530 x1876217315783168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789300412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3769.847379] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3769.903112] Lustre: 120154:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3770.442018] LustreError: 123476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300413 with bad export cookie 16994592314596018029 [ 3770.444197] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.461045] LustreError: 123476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3771.114768] Lustre: server umount lustre-MDT0001 complete [ 3780.989804] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3784.002661] Key type lgssc unregistered [ 3784.344784] LNet: 125732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3784.352053] LNetError: 125732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3784.374631] LNet: Removed LNI 192.168.202.138@tcp [ 3785.551512] Key type .llcrypt unregistered [ 3785.553798] Key type ._llcrypt unregistered [ 3817.532530] Key type ._llcrypt registered [ 3817.534898] Key type .llcrypt registered [ 3817.646553] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3818.499707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3818.524372] alg: No test for adler32 (adler32-zlib) [ 3819.744928] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3820.086408] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3821.799146] Key type lgssc registered [ 3823.454676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3832.544766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3832.567409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.152533] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3834.195018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.202184] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.38/24@tcp [ 3839.631216] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3844.579489] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.608843] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3848.853694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.233623] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3849.266060] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 3849.317763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.994814] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3860.777569] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3867.519427] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.452087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.761828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.909213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3881.737576] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3889.698113] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3903.976763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3903.986950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.015842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3904.481221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3904.498863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.600885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.616922] Lustre: Skipped 3 previous similar messages [ 3910.060092] Lustre: server umount lustre-OST0000 complete [ 3914.726246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.750577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.755829] Lustre: Skipped 1 previous similar message [ 3919.840940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.853844] Lustre: Skipped 1 previous similar message [ 3920.429570] Lustre: server umount lustre-MDT0000 complete [ 3923.765473] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300566 with bad export cookie 6866750629731990891 [ 3923.772407] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.774580] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3924.048943] Lustre: server umount lustre-MDT0001 complete [ 3933.495869] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3935.775342] Key type lgssc unregistered [ 3936.094756] LNet: 130877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3936.099553] LNetError: 130877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3936.117132] LNet: Removed LNI 192.168.202.138@tcp [ 3937.004080] Key type .llcrypt unregistered [ 3937.008151] Key type ._llcrypt unregistered [ 3961.351704] Key type ._llcrypt registered [ 3961.353874] Key type .llcrypt registered [ 3961.442547] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 3978.681835] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3979.777481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3979.834208] alg: No test for adler32 (adler32-zlib) [ 3980.848342] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 3981.140499] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3982.824080] Key type lgssc registered [ 3984.208310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4021.018439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4021.059443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.433813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4022.479807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4022.600094] Lustre: lustre-MDT0000: new disk, initializing [ 4022.683767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.721990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4027.952867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4038.318394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.423686] Lustre: 134278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4038.469320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4038.471630] Lustre: Skipped 1 previous similar message [ 4038.522981] Lustre: lustre-MDT0001: new disk, initializing [ 4038.612145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4038.649814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4038.663781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4043.073125] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4052.089095] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4058.136978] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4064.304757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4064.605385] Lustre: lustre-OST0000: new disk, initializing [ 4064.613684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4064.630375] Lustre: 136244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4064.728908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.827759] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4071.001322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4071.032859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4071.275440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4081.143564] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4086.602990] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4086.846321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4092.018851] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4092.234653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4096.994934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.995252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.040119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4097.047830] Lustre: Skipped 1 previous similar message [ 4100.483376] Lustre: server umount lustre-OST0000 complete [ 4105.184799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.201828] LustreError: Skipped 1 previous similar message [ 4105.216589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.242606] Lustre: Skipped 1 previous similar message [ 4105.251753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.233590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.235220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.734792] Lustre: server umount lustre-MDT0000 complete [ 4114.077221] LustreError: 134269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300757 with bad export cookie 14826531360025680063 [ 4114.082180] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.098730] LustreError: 134269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4114.564923] Lustre: server umount lustre-MDT0001 complete [ 4123.236817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.705296] LustreError: 138460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.725134] LustreError: 138460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4123.810554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.480411] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4129.255647] LustreError: 138461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.351180] LustreError: 138460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.274435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.730810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.956480] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4150.766233] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4156.999355] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4163.975263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4164.345336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.289326] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4178.746549] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4183.936145] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4184.171564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4189.311001] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4189.503091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4197.997762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.167853] Lustre: lustre-OST0001: new disk, initializing [ 4198.172152] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4198.177285] Lustre: 141832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4198.277823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4205.288849] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4208.249253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4208.257048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4208.357350] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4212.216203] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4212.496774] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4217.644265] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4217.894333] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4224.294176] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff889185029000.ost_server_uuid 50 [ 4226.125669] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff889185029000.ost_server_uuid in FULL state after 0 sec [ 4236.667502] Lustre: server umount lustre-OST0001 complete [ 4239.328385] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4239.346757] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.348447] LustreError: Skipped 1 previous similar message [ 4239.360601] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.382957] Lustre: Skipped 1 previous similar message [ 4239.424934] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4244.460257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.461098] LustreError: 140285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.481559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4244.481570] Lustre: Skipped 1 previous similar message [ 4244.484827] Lustre: Skipped 1 previous similar message [ 4244.493611] LustreError: 140285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4244.526089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4249.570453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4249.572832] Lustre: Skipped 2 previous similar messages [ 4254.704826] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.735170] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4255.200572] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4255.547555] Lustre: server umount lustre-OST0000 complete [ 4259.813028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.832720] Lustre: Skipped 1 previous similar message [ 4259.835947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.842024] Lustre: Skipped 3 previous similar messages [ 4273.119587] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4273.300984] Lustre: server umount lustre-MDT0000 complete [ 4275.168337] LustreError: 138460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.187287] LustreError: 138460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4277.271530] LustreError: 138440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300920 with bad export cookie 14826531360025680945 [ 4277.274561] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.285791] LustreError: 138440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4277.880156] Lustre: server umount lustre-MDT0001 complete [ 4298.335491] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4301.258783] Key type lgssc unregistered [ 4301.487139] LNet: 144849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4301.492904] LNetError: 144849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4302.569797] LNet: Removed LNI 192.168.202.138@tcp [ 4303.489348] Key type .llcrypt unregistered [ 4303.491967] Key type ._llcrypt unregistered [ 4342.258394] Key type ._llcrypt registered [ 4342.262036] Key type .llcrypt registered [ 4342.401720] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4360.820905] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4362.710039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4362.738027] alg: No test for adler32 (adler32-zlib) [ 4363.796089] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 4364.018355] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4365.743930] Key type lgssc registered [ 4367.153323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4408.143561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4408.188961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.513431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4409.542625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4409.619942] Lustre: lustre-MDT0000: new disk, initializing [ 4409.693747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.707965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4414.117533] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4427.136978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.313122] Lustre: 149026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4427.366452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4427.370109] Lustre: Skipped 1 previous similar message [ 4427.443274] Lustre: lustre-MDT0001: new disk, initializing [ 4427.571850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4427.621475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4427.629238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4432.448194] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4442.903710] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.581646] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4457.202402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4457.487466] Lustre: lustre-OST0000: new disk, initializing [ 4457.494796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4457.502385] Lustre: 150995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4457.580715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.660231] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4463.699355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4463.715638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4463.930661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4476.251422] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4482.468513] Lustre: Failing over lustre-OST0000 [ 4482.651815] Lustre: server umount lustre-OST0000 complete [ 4500.831163] Lustre: 147250:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301127/real 1789301127] req@0000000052247177 x1876218092750464/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789301143 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4500.896615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.478461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.976078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.986506] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4504.975913] Lustre: 147249:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301132/real 1789301132] req@000000002223381f x1876218092751360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789301148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4505.027936] Lustre: 147249:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4507.990899] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4510.033262] Lustre: 147249:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301137/real 1789301137] req@00000000c0b10754 x1876218092752384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789301153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4510.065871] Lustre: 147249:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4510.637257] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4511.109465] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4511.113621] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4511.794696] Lustre: Failing over lustre-MDT0000 [ 4512.340827] Lustre: server umount lustre-MDT0000 complete [ 4514.797255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4514.808975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.834513] Lustre: Skipped 1 previous similar message [ 4515.871184] Lustre: 147247:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301143/real 1789301143] req@000000006f3ec09c x1876218092753152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789301159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4516.333342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.857464] LustreError: 149033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.869097] LustreError: 149033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4521.440755] LustreError: 150751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.450860] LustreError: 150751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4525.985452] LustreError: 149032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.610928] Lustre: 149028:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301155/real 1789301155] req@0000000094941f5d x1876218092762112/t0(0) o101->MGC192.168.202.138@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1789301171 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4528.643878] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4528.670270] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.683459] Lustre: 149028:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4528.695171] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4532.869439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.731058] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.760033] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4538.849885] LustreError: 147246:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007ca7b258 x1876218092767744/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4539.079626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.114388] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4542.360599] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4542.742274] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4544.505869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4544.514033] Lustre: Skipped 1 previous similar message [ 4544.546462] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4544.599379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4552.160319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.169040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.178189] Lustre: Skipped 1 previous similar message [ 4552.184336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.744222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.752058] Lustre: Skipped 1 previous similar message [ 4557.127791] Lustre: server umount lustre-OST0000 complete [ 4564.964458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.978870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4564.979285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.991629] Lustre: Skipped 2 previous similar messages [ 4566.941595] Lustre: server umount lustre-MDT0000 complete [ 4570.093358] LustreError: 150109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.110686] LustreError: 150109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4570.713382] LustreError: 149018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301213 with bad export cookie 7333492510465792680 [ 4570.723277] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4570.729671] LustreError: 149018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4571.113437] Lustre: server umount lustre-MDT0001 complete [ 4584.297308] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4587.826572] Key type lgssc unregistered [ 4588.218595] LNet: 154650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4588.222633] LNetError: 154650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4588.241325] LNet: Removed LNI 192.168.202.138@tcp [ 4589.304176] Key type .llcrypt unregistered [ 4589.306988] Key type ._llcrypt unregistered [ 4627.063673] Key type ._llcrypt registered [ 4627.066223] Key type .llcrypt registered [ 4627.160536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4628.429676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4628.754570] alg: No test for adler32 (adler32-zlib) [ 4629.971877] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 4630.301920] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4632.097977] Key type lgssc registered [ 4633.394351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4641.641871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4641.676334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.275480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.453178] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4653.537072] LustreError: 156166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.551114] LustreError: 156166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4657.184439] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4658.658923] LustreError: 156165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.053348] LustreError: 156194:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4661.060434] LustreError: 156194:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4661.070654] LustreError: 156194:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4661.402123] Lustre: server umount lustre-MDT0000 complete [ 4677.287897] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4687.499878] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4725.623408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.836167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.862206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.949511] Lustre: lustre-MDT0000: new disk, initializing [ 4726.016466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.031154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4730.563328] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4741.886265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.042657] Lustre: 159501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4742.103331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4742.107469] Lustre: Skipped 1 previous similar message [ 4742.191402] Lustre: lustre-MDT0001: new disk, initializing [ 4742.288301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4742.346676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4742.361311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.267552] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4757.753171] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4763.856189] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4771.086564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.484856] Lustre: lustre-OST0000: new disk, initializing [ 4771.493836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4771.504968] Lustre: 161470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4771.658245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.051578] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4780.650768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4780.681671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4780.809137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.991408] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4792.707700] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4794.265297] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:10:36 (1789301436) [ 5021.663820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.665034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5021.671747] Lustre: Skipped 1 previous similar message [ 5025.923560] Lustre: server umount lustre-MDT0000 complete [ 5026.784112] LustreError: 159511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.800563] LustreError: 159511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5028.985749] LustreError: 159492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301672 with bad export cookie 12609055191918035257 [ 5028.987696] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.998296] LustreError: 159492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5031.903985] LustreError: 159507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.908283] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.939398] Lustre: Skipped 1 previous similar message [ 5031.944863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5031.948197] Lustre: Skipped 2 previous similar messages [ 5033.952901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5035.371399] Lustre: server umount lustre-MDT0001 complete [ 5045.510512] Lustre: server umount lustre-OST0000 complete [ 5056.690525] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5067.565787] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5107.124391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.368839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5107.420722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5107.524359] Lustre: lustre-MDT0000: new disk, initializing [ 5107.629579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.646470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5112.557787] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5123.526367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.648409] Lustre: 166129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5123.708642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5123.712953] Lustre: Skipped 1 previous similar message [ 5123.788661] Lustre: lustre-MDT0001: new disk, initializing [ 5123.938323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5124.022149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5124.042213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5130.182711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5141.469587] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5148.817702] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5157.570759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5157.857353] Lustre: lustre-OST0000: new disk, initializing [ 5157.867025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5157.873394] Lustre: 168099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5157.977790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5163.655472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5163.673544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5163.841607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5166.519428] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5176.408769] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5181.690880] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5181.933897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5188.747678] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5189.018306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5194.724000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.724845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5194.733885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.742628] Lustre: Skipped 1 previous similar message [ 5198.441225] Lustre: server umount lustre-OST0000 complete [ 5204.961080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.965297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.984163] Lustre: Skipped 1 previous similar message [ 5205.000237] Lustre: Skipped 2 previous similar messages [ 5209.813850] Lustre: server umount lustre-MDT0000 complete [ 5210.094379] LustreError: 166141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.130671] LustreError: 166141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5213.951952] LustreError: 166122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301857 with bad export cookie 12609055191918194087 [ 5213.956464] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5213.966813] LustreError: 166122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5214.473872] Lustre: server umount lustre-MDT0001 complete [ 5224.394584] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:17:46 (1789301866) [ 5240.893891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5250.907828] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5285.105513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.330281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5285.343309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5285.391165] Lustre: lustre-MDT0000: new disk, initializing [ 5285.447064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5285.460145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5290.576891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5301.228596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.368748] Lustre: 172858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.676925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5306.140729] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5315.800482] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5320.629692] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5328.894226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5329.346863] Lustre: lustre-OST0000: new disk, initializing [ 5329.349783] Lustre: Skipped 1 previous similar message [ 5329.352651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5329.359401] Lustre: Skipped 2 previous similar messages [ 5329.364296] Lustre: 174837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5336.679759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5336.699934] Lustre: Skipped 1 previous similar message [ 5336.708597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5336.856592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5337.949965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5347.828940] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5357.536224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.553897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.559261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.575560] Lustre: Skipped 1 previous similar message [ 5361.674737] Lustre: server umount lustre-OST0000 complete [ 5367.776980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.787897] Lustre: Skipped 1 previous similar message [ 5372.031336] Lustre: server umount lustre-MDT0000 complete [ 5372.899259] LustreError: 172868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5372.918916] LustreError: 172868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5375.507299] LustreError: 172849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302018 with bad export cookie 12609055191918194969 [ 5375.511851] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.524424] LustreError: 172849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5375.887638] Lustre: server umount lustre-MDT0001 complete [ 5384.780367] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:20:26 (1789302026) [ 5401.242033] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5412.395493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5449.103562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.374144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5449.471739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5449.581300] Lustre: lustre-MDT0000: new disk, initializing [ 5449.694231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.701970] Lustre: Skipped 2 previous similar messages [ 5449.719897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5452.355216] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5463.064472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.301751] Lustre: 179160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5463.576944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5468.773489] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5479.657309] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5486.170613] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5493.018655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.475374] Lustre: 181138:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5500.592371] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5503.648590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5503.789570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5510.852733] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5519.328429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.346359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.348138] LustreError: Skipped 1 previous similar message [ 5519.351461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.371850] Lustre: Skipped 1 previous similar message [ 5519.400232] Lustre: Skipped 4 previous similar messages [ 5524.371557] Lustre: server umount lustre-OST0000 complete [ 5534.746451] LustreError: 179173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.793651] Lustre: server umount lustre-MDT0000 complete [ 5538.834775] LustreError: 179154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302182 with bad export cookie 12609055191918196047 [ 5538.843933] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5538.850470] LustreError: 179154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5539.192090] Lustre: server umount lustre-MDT0001 complete [ 5551.400831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5554.478754] Key type lgssc unregistered [ 5554.870082] LNet: 182995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5554.878542] LNetError: 182995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5555.947749] LNet: Removed LNI 192.168.202.138@tcp [ 5556.861648] Key type .llcrypt unregistered [ 5556.864343] Key type ._llcrypt unregistered [ 5581.229929] Key type ._llcrypt registered [ 5581.231597] Key type .llcrypt registered [ 5581.443552] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5582.843403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5582.936351] alg: No test for adler32 (adler32-zlib) [ 5584.131505] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 5584.460956] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5586.127257] Key type lgssc registered [ 5587.451594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5595.870221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5595.906512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5597.382880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.668553] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5606.882648] LustreError: 184413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5606.902433] LustreError: 184413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5611.255428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.679103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5616.775463] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5624.311989] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5630.551485] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.090485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.494191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.626381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5644.793936] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5651.608223] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5676.513085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.520607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5676.547729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5676.571951] Lustre: Skipped 1 previous similar message [ 5680.643561] Lustre: server umount lustre-OST0000 complete [ 5686.756593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.788542] Lustre: Skipped 2 previous similar messages [ 5686.793465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.804971] Lustre: Skipped 1 previous similar message [ 5691.819577] Lustre: server umount lustre-MDT0000 complete [ 5691.875608] LustreError: 185141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.972434] LustreError: 184392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302339 with bad export cookie 5100201214919897231 [ 5695.984273] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.003477] LustreError: 184392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5696.683459] Lustre: server umount lustre-MDT0001 complete [ 5707.670292] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5710.643990] Key type lgssc unregistered [ 5710.924536] LNet: 187966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5710.929192] LNetError: 187966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5710.941770] LNet: Removed LNI 192.168.202.138@tcp [ 5711.857463] Key type .llcrypt unregistered [ 5711.859691] Key type ._llcrypt unregistered [ 5735.337184] Key type ._llcrypt registered [ 5735.343065] Key type .llcrypt registered [ 5735.462180] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5736.762160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5736.826744] alg: No test for adler32 (adler32-zlib) [ 5737.877648] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 5738.124084] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5739.799224] Key type lgssc registered [ 5740.849634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5748.015457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5748.049287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.475526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.991318] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5758.949870] LustreError: 189381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.969275] LustreError: 189381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5760.344794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.655944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5764.752341] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5772.489952] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5777.637225] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5782.487817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.770717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.086082] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5794.099612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5797.938446] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5804.517120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.520861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.539707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.543318] Lustre: Skipped 1 previous similar message [ 5809.505321] Lustre: server umount lustre-OST0000 complete [ 5816.668768] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff889187aba800.ost_server_uuid 50 [ 5818.075973] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5832.160505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.165221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.182221] Lustre: Skipped 1 previous similar message [ 5832.190402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.204426] Lustre: Skipped 1 previous similar message [ 5834.210249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.212673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.248210] Lustre: Skipped 1 previous similar message [ 5838.258486] Lustre: server umount lustre-MDT0000 complete [ 5839.329443] LustreError: 189381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.353742] LustreError: 189381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5842.046773] LustreError: 189361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302485 with bad export cookie 15875476918945161290 [ 5842.048524] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.054280] LustreError: 189361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5842.445687] Lustre: server umount lustre-MDT0001 complete [ 5850.614435] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:28:12 (1789302492) [ 5856.690021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.123305] LustreError: 193294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.195418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5861.515487] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5862.372716] LustreError: 193295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.496064] LustreError: 193294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.207561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.602741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5874.579906] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5881.491671] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5886.640809] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5893.020389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.531309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.023250] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5903.801774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5905.353955] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5913.115456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.408495] Lustre: lustre-OST0001: new disk, initializing [ 5913.417309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5913.426474] Lustre: 196135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5913.574337] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5920.464291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5920.487708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5920.625593] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5922.120913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5932.205706] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5936.105785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.121162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.232829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.241942] Lustre: Skipped 3 previous similar messages [ 5941.382664] Lustre: server umount lustre-OST0000 complete [ 5943.197139] LustreError: 195116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.314894] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5948.630630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5953.566398] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5953.743164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5961.709959] LustreError: 195116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.737828] LustreError: 195116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5971.941687] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5971.946699] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.947027] LustreError: Skipped 1 previous similar message [ 5971.950530] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5971.969298] Lustre: Skipped 2 previous similar messages [ 5974.167477] Lustre: server umount lustre-OST0001 complete [ 5978.082654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.181394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.190952] Lustre: Skipped 1 previous similar message [ 5984.084467] Lustre: server umount lustre-MDT0000 complete [ 5987.618422] LustreError: 193276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302630 with bad export cookie 15875476918945162361 [ 5987.631563] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.635947] LustreError: 193276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5988.068824] Lustre: server umount lustre-MDT0001 complete [ 6010.322775] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:30:52 (1789302652) [ 6017.528990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.942561] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6018.027440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6018.351418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.194924] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6030.894452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.975118] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6031.005949] Lustre: 199575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6036.625068] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6044.680509] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6050.677275] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6057.982242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6058.226191] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6063.780609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6065.838359] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6074.268295] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6082.294821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.423793] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6082.811869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.826422] Lustre: Skipped 2 previous similar messages [ 6089.440810] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6097.875499] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6103.479540] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 6103.487989] Lustre: Skipped 4 previous similar messages [ 6104.546303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.570384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.580253] Lustre: server umount lustre-OST0000 complete [ 6108.582776] LustreError: 201708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.595186] LustreError: 201708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6124.513269] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.552825] Lustre: server umount lustre-OST0001 complete [ 6138.212484] Lustre: server umount lustre-MDT0000 complete [ 6141.993802] LustreError: 199567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302785 with bad export cookie 15875476918945163768 [ 6141.996170] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6142.008352] LustreError: 199567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6142.636572] Lustre: server umount lustre-MDT0001 complete [ 6170.903955] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:33:32 (1789302812) [ 6189.266932] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6200.759887] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6234.700242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.963384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.026336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6235.134545] Lustre: lustre-MDT0000: new disk, initializing [ 6235.268224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.284819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6239.320608] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6248.828186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.003990] Lustre: 207675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6249.048783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6249.051923] Lustre: Skipped 1 previous similar message [ 6249.119693] Lustre: lustre-MDT0001: new disk, initializing [ 6249.232540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6249.243457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6253.880234] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6262.489632] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6267.633669] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6274.231991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.470652] Lustre: lustre-OST0000: new disk, initializing [ 6274.475514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6274.483052] Lustre: 209641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6282.179001] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6283.398253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6283.412086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6283.573307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6292.535441] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6296.689347] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6296.948776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6300.982982] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6301.281752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6304.233126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.234787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.241958] LustreError: Skipped 2 previous similar messages [ 6304.245655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.255400] Lustre: Skipped 6 previous similar messages [ 6304.275906] Lustre: Skipped 9 previous similar messages [ 6309.190836] Lustre: server umount lustre-OST0000 complete [ 6327.263126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6327.455416] Lustre: server umount lustre-MDT0000 complete [ 6329.826820] LustreError: 207686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.845535] LustreError: 207686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6331.006995] LustreError: 207667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302974 with bad export cookie 15875476918945165329 [ 6331.017926] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.029919] LustreError: 207667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6331.403906] Lustre: server umount lustre-MDT0001 complete [ 6339.489546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.870668] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6353.088643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.035956] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6366.179502] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.286771] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6379.799373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.999721] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6398.304622] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.843545] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6403.021492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6406.416814] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6406.560952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6409.186887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.191922] Lustre: Skipped 1 previous similar message [ 6409.195729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.195806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.201628] Lustre: Skipped 7 previous similar messages [ 6414.952517] Lustre: server umount lustre-OST0000 complete [ 6421.486591] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6426.466706] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6455.344233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6461.983698] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6463.870354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.890317] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.897338] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6463.929055] Lustre: Skipped 1 previous similar message [ 6469.223550] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.221151] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6484.244539] Lustre: server umount lustre-OST0000 complete [ 6502.367867] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6502.558566] Lustre: server umount lustre-MDT0000 complete [ 6504.935618] LustreError: 211855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.956416] LustreError: 211855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6506.163644] LustreError: 211841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303149 with bad export cookie 15875476918945166211 [ 6506.179067] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.185154] LustreError: 211841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6506.440732] Lustre: server umount lustre-MDT0001 complete [ 6514.974699] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:39:16 (1789303156) [ 6521.662465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.154423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.158533] Lustre: Skipped 6 previous similar messages [ 6525.982468] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6535.123923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.660266] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6549.299134] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6555.452236] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.945415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.328796] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6577.677043] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6582.302334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6582.543328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6586.564341] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6586.804690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6593.842940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.058413] Lustre: lustre-OST0001: new disk, initializing [ 6594.064863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6594.070209] Lustre: 221204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6599.750655] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6602.828130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6602.845088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6602.945383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6607.910454] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6612.824095] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6613.054207] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6617.562205] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6617.723451] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6623.722441] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6623.743382] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.752396] LustreError: Skipped 1 previous similar message [ 6623.770481] Lustre: Skipped 5 previous similar messages [ 6623.786196] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6623.807779] Lustre: Skipped 11 previous similar messages [ 6625.414621] Lustre: server umount lustre-OST0001 complete [ 6631.418954] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6631.711355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6635.930533] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6636.204624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6665.396759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.337989] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6671.358823] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6671.371514] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6673.092645] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6679.465083] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.965973] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6688.815643] Lustre: server umount lustre-OST0001 complete [ 6712.032448] Lustre: server umount lustre-OST0000 complete [ 6721.921097] Lustre: server umount lustre-MDT0000 complete [ 6726.452714] LustreError: 224774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303369 with bad export cookie 15875476918945167177 [ 6726.459636] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.464282] LustreError: 224774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6732.839579] Lustre: server umount lustre-MDT0001 complete [ 6756.148842] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:43:18 (1789303398) [ 6762.180611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.391288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6762.408967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6766.448356] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6773.987352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.071355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6774.103916] Lustre: 226447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6778.502582] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6785.715541] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6790.869305] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6797.054180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.235120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6803.410887] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6811.482367] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.840423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6819.888346] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6825.124852] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6832.728693] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.187876] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6839.747868] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6839.970864] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6844.794480] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6845.067443] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6851.404665] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff889198dd2000.ost_server_uuid 50 [ 6853.230542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff889198dd2000.ost_server_uuid in IDLE state after 0 sec [ 6854.328966] Lustre: Permanently deactivating lustre-OST0001 [ 6854.360446] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6858.067220] Lustre: Permanently reactivating lustre-OST0001 [ 6864.140301] Lustre: server umount lustre-OST0001 complete [ 6865.893916] LustreError: 228295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.919201] LustreError: 228295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 6869.986786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6869.994050] LustreError: Skipped 3 previous similar messages [ 6874.186982] Lustre: server umount lustre-OST0000 complete [ 6881.249145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.251170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6881.259379] Lustre: Skipped 11 previous similar messages [ 6881.275450] Lustre: Skipped 14 previous similar messages [ 6884.336669] Lustre: server umount lustre-MDT0000 complete [ 6887.836650] LustreError: 227188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303531 with bad export cookie 15875476918945168458 [ 6887.841329] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.848101] LustreError: 227188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6888.102894] Lustre: server umount lustre-MDT0001 complete [ 6910.498795] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:45:52 (1789303552) [ 6918.182778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.621618] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6918.649355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6918.658155] Lustre: Skipped 5 previous similar messages [ 6925.238325] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6934.246947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.395121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6934.452704] Lustre: 233140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6938.951361] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6946.546782] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6951.557621] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6958.102589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.223469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6958.235228] Lustre: Permanently deactivating lustre-OST0000 [ 6958.242292] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6965.827588] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6966.821192] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6974.172292] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.837330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6981.980284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6989.083306] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6991.070829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6997.194574] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.811749] Lustre: Permanently reactivating lustre-OST0000 [ 6998.861412] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6998.876039] Lustre: Skipped 2 previous similar messages [ 6999.324593] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6999.333124] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000bedc4d81, cur 1789303642 deadline 1789303640 last 1789303610 [ 6999.389136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7015.855568] Lustre: server umount lustre-OST0001 complete [ 7025.614369] Lustre: server umount lustre-OST0000 complete [ 7035.423452] Lustre: server umount lustre-MDT0000 complete [ 7038.786523] LustreError: 235020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303681 with bad export cookie 15875476918945170145 [ 7038.787728] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.795832] LustreError: 235020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7039.049496] Lustre: server umount lustre-MDT0001 complete [ 7050.028405] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7053.374651] Key type lgssc unregistered [ 7053.720776] LNet: 237997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7053.734157] LNetError: 237997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7053.759802] LNet: Removed LNI 192.168.202.138@tcp [ 7054.598150] Key type .llcrypt unregistered [ 7054.599970] Key type ._llcrypt unregistered [ 7077.805993] Key type ._llcrypt registered [ 7077.807754] Key type .llcrypt registered [ 7077.898552] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7079.000119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7079.105866] alg: No test for adler32 (adler32-zlib) [ 7080.166410] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7080.446800] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7082.103566] Key type lgssc registered [ 7083.645279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7093.176477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7093.246735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.894542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.300190] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7105.512451] LustreError: 239461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.531037] LustreError: 239461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7106.617744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.757357] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7106.789098] Lustre: 239449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7106.803595] Lustre: Permanently deactivating lustre-MDT0001 [ 7106.813763] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7107.096434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.090684] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7114.244119] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7119.071691] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7124.592908] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7130.977256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.238983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.283756] LustreError: 241305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.299705] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7137.274505] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7137.774590] LustreError: 241302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.840160] LustreError: 241305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.875974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7143.909579] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.956303] LustreError: 241302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.976753] LustreError: 241302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7150.180626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.546222] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7155.708202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7155.732214] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7157.949773] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7165.868398] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7170.197677] Lustre: Permanently reactivating lustre-MDT0001 [ 7170.247307] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7170.258277] Lustre: Skipped 2 previous similar messages [ 7174.591304] Lustre: lustre-MDT0001: Client be2dd2f8-0907-459d-9efb-fe13585951c4 (at 192.168.202.38@tcp) reconnecting [ 7178.925816] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.953857] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7178.968408] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7178.979502] LustreError: 243170:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7178.993628] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7178.998897] LustreError: 243170:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7178.999091] LustreError: 243170:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.138@tcp: cfg command failed: rc = -108 [ 7178.999115] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7178.999115] [ 7178.999462] LustreError: 239452:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7180.709346] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7182.921105] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7191.220561] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7191.642460] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7198.600031] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7198.868979] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7201.836301] Lustre: Permanently deactivating lustre-MDT0001 [ 7201.862448] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7201.866630] Lustre: Skipped 2 previous similar messages [ 7211.154303] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7224.292256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.295244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7224.299882] Lustre: Skipped 3 previous similar messages [ 7224.322279] Lustre: Skipped 3 previous similar messages [ 7226.986811] Lustre: server umount lustre-MDT0000 complete [ 7229.409894] LustreError: 239460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.425426] LustreError: 239460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7230.578412] LustreError: 243006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303873 with bad export cookie 11140902179349650311 [ 7230.579396] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.594292] LustreError: 243006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7231.049475] Lustre: server umount lustre-MDT0001 complete [ 7237.055863] Lustre: server umount lustre-OST0000 complete [ 7248.594955] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:51:30 (1789303890) [ 7262.199769] Lustre: server umount lustre-OST0001 complete [ 7270.180132] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7279.947905] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7314.294875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.514788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7314.523481] Lustre: Skipped 2 previous similar messages [ 7314.543595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7314.622598] Lustre: lustre-MDT0000: new disk, initializing [ 7314.671849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.708817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7318.875522] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7329.210329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.304570] Lustre: 247769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7329.345263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7329.350257] Lustre: Skipped 1 previous similar message [ 7329.404606] Lustre: lustre-MDT0001: new disk, initializing [ 7329.470327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.495318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7329.507578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7334.419455] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7344.361440] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7349.947459] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.279494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.626453] Lustre: lustre-OST0000: new disk, initializing [ 7357.632765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7357.642649] Lustre: 249737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7357.767030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.005382] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7366.189953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7366.206832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7366.286042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7373.478368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7375.356220] LustreError: 249270:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7384.700393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.954732] Lustre: lustre-OST0001: new disk, initializing [ 7384.959713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7384.982059] Lustre: 250851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7385.389371] LustreError: 249270:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7385.443807] LustreError: 247776:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7392.089517] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7392.928385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7392.961700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7393.197574] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7395.457880] LustreError: 247776:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7402.936619] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.337252] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7408.623159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7408.628818] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.632570] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7408.643517] Lustre: Skipped 1 previous similar message [ 7413.737752] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7413.753192] Lustre: Skipped 2 previous similar messages [ 7414.207946] Lustre: server umount lustre-OST0001 complete [ 7418.888807] LustreError: 249721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.911266] LustreError: 249721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7433.187851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.188934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.204889] LustreError: Skipped 1 previous similar message [ 7433.209658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.246540] Lustre: Skipped 1 previous similar message [ 7433.279579] Lustre: Skipped 1 previous similar message [ 7438.338239] Lustre: server umount lustre-OST0000 complete [ 7444.452091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.470710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.478756] Lustre: Skipped 3 previous similar messages [ 7449.858767] Lustre: server umount lustre-MDT0000 complete [ 7453.153419] LustreError: 250850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304096 with bad export cookie 11140902179349653118 [ 7453.158040] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7453.164044] LustreError: 250850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7459.656316] Lustre: server umount lustre-MDT0001 complete [ 7469.087060] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7472.484829] Key type lgssc unregistered [ 7472.848785] LNet: 252886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7472.852607] LNetError: 252886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7472.862709] LNet: Removed LNI 192.168.202.138@tcp [ 7473.876704] Key type .llcrypt unregistered [ 7473.878390] Key type ._llcrypt unregistered [ 7518.719428] Key type ._llcrypt registered [ 7518.721767] Key type .llcrypt registered [ 7518.833123] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7520.761375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7520.932000] alg: No test for adler32 (adler32-zlib) [ 7522.062503] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7522.293768] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7523.959170] Key type lgssc registered [ 7525.167812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7532.226955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7532.243062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.457238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7533.474210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7533.630934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7538.162767] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7546.761691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.857987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7546.904341] Lustre: 255090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7547.194564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.137990] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7557.925472] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7563.987490] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7572.145077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7572.289785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7572.387136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.291758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7578.822238] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7586.907496] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7644.642408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.647720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.665910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.158142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7645.168640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.184496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.189690] Lustre: Skipped 1 previous similar message [ 7646.467000] Lustre: server umount lustre-OST0000 complete [ 7650.337241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7663.250057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7663.798348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.849834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7665.861974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7669.992930] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7670.021735] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7671.236639] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7678.865160] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7721.442259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.452402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.458363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.469109] Lustre: Skipped 1 previous similar message [ 7721.489791] Lustre: Skipped 1 previous similar message [ 7725.359627] Lustre: server umount lustre-OST0000 complete [ 7731.681187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7731.682861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.690349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.710237] Lustre: Skipped 1 previous similar message [ 7731.722673] Lustre: Skipped 1 previous similar message [ 7735.532310] Lustre: server umount lustre-MDT0000 complete [ 7739.546763] LustreError: 255084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304382 with bad export cookie 16626768489749896360 [ 7739.548585] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.557560] LustreError: 255084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7740.051634] Lustre: server umount lustre-MDT0001 complete [ 7751.920202] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7754.788151] Key type lgssc unregistered [ 7755.118620] LNet: 260515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7755.141948] LNetError: 260515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7755.157768] LNet: Removed LNI 192.168.202.138@tcp [ 7756.126181] Key type .llcrypt unregistered [ 7756.132382] Key type ._llcrypt unregistered [ 7777.917893] Key type ._llcrypt registered [ 7777.919709] Key type .llcrypt registered [ 7777.994394] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7778.819258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7778.866911] alg: No test for adler32 (adler32-zlib) [ 7779.926991] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7780.103174] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7781.727221] Key type lgssc registered [ 7782.757144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7790.071966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7790.091889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.759085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.306088] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7801.323753] LustreError: 261922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.347163] LustreError: 261922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7804.110133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.355963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7807.845544] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7814.172196] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7818.946718] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7825.194664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.476919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.575950] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7836.746364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7837.284282] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7860.201459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.207305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.216028] Lustre: Skipped 1 previous similar message [ 7860.238550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7865.315711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7865.331213] Lustre: Skipped 2 previous similar messages [ 7865.514937] Lustre: server umount lustre-OST0000 complete [ 7870.434311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.445844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.450742] Lustre: Skipped 1 previous similar message [ 7874.848073] Lustre: server umount lustre-MDT0000 complete [ 7875.554869] LustreError: 261918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.585872] LustreError: 261918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7878.425797] LustreError: 265282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304521 with bad export cookie 12365411492667580154 [ 7878.442813] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7878.752424] Lustre: server umount lustre-MDT0001 complete [ 7888.914564] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7892.142915] Key type lgssc unregistered [ 7892.534152] LNet: 266234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7892.543313] LNetError: 266234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7892.573947] LNet: Removed LNI 192.168.202.138@tcp [ 7893.606989] Key type .llcrypt unregistered [ 7893.614249] Key type ._llcrypt unregistered [ 7910.628902] Key type ._llcrypt registered [ 7910.636988] Key type .llcrypt registered [ 7910.750041] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7912.068314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7912.227604] alg: No test for adler32 (adler32-zlib) [ 7913.285383] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 7913.636340] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7915.303180] Key type lgssc registered [ 7916.411140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7924.508214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7924.536559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.978449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.969558] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7936.492250] LustreError: 267464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.525425] LustreError: 267464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7939.227538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.487951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7944.408359] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7951.591384] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7957.265678] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7963.749340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.030761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.665613] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7972.228617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7979.107504] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.808102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.810348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7992.819536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.841248] Lustre: Skipped 1 previous similar message [ 7997.167894] Lustre: server umount lustre-OST0000 complete [ 8003.043897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.046724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8003.068707] Lustre: Skipped 1 previous similar message [ 8003.083197] Lustre: Skipped 2 previous similar messages [ 8007.478901] Lustre: server umount lustre-MDT0000 complete [ 8008.176448] LustreError: 267464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.199603] LustreError: 267464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8010.814129] LustreError: 267446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304653 with bad export cookie 15570205125537383783 [ 8010.821362] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8010.828310] LustreError: 267446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8011.275890] Lustre: server umount lustre-MDT0001 complete [ 8021.946602] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8024.930531] Key type lgssc unregistered [ 8025.234058] LNet: 271138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8025.240500] LNetError: 271138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8026.282527] LNet: Removed LNI 192.168.202.138@tcp [ 8027.336821] Key type .llcrypt unregistered [ 8027.343935] Key type ._llcrypt unregistered [ 8049.684182] Key type ._llcrypt registered [ 8049.686392] Key type .llcrypt registered [ 8049.874099] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8051.119850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8051.150609] alg: No test for adler32 (adler32-zlib) [ 8052.186780] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 8052.401538] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8054.105766] Key type lgssc registered [ 8055.205291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8061.931594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8061.950827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.521177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.099769] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8073.702467] LustreError: 272553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.715235] LustreError: 272553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8076.690952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.083900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8081.595897] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8089.119254] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8094.846299] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8101.936690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8102.459646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8108.377983] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8108.593135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8115.397949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8139.237685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.251708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.263741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.287660] Lustre: Skipped 1 previous similar message [ 8143.005434] Lustre: server umount lustre-OST0000 complete [ 8148.961673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8148.967628] LustreError: Skipped 1 previous similar message [ 8148.972378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8148.983872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8148.995585] Lustre: Skipped 1 previous similar message [ 8153.343867] Lustre: server umount lustre-MDT0000 complete [ 8154.594958] LustreError: 273283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.616811] LustreError: 273283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8157.056511] LustreError: 272533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304800 with bad export cookie 15073197067529216819 [ 8157.084260] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.530790] Lustre: server umount lustre-MDT0001 complete [ 8169.900882] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8172.919992] Key type lgssc unregistered [ 8173.302584] LNet: 276856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8173.313241] LNetError: 276856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8173.333668] LNet: Removed LNI 192.168.202.138@tcp [ 8174.233621] Key type .llcrypt unregistered [ 8174.236995] Key type ._llcrypt unregistered [ 8193.241413] Key type ._llcrypt registered [ 8193.244773] Key type .llcrypt registered [ 8193.383865] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8194.948164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8194.995427] alg: No test for adler32 (adler32-zlib) [ 8196.092933] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 8196.376929] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8198.127275] Key type lgssc registered [ 8199.731671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8208.604811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8208.620035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.191408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.319224] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8220.647254] LustreError: 278087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.669878] LustreError: 278087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8223.297988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.686161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8228.381800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8236.017039] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8242.549141] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8248.989730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.466646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.602878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8255.745319] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8263.367553] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8276.449751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8276.455776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.468926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8278.499741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8278.503502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.504898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.813360] Lustre: server umount lustre-OST0000 complete [ 8288.743173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.746898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.764576] Lustre: Skipped 1 previous similar message [ 8288.775221] Lustre: Skipped 2 previous similar messages [ 8292.584822] Lustre: server umount lustre-MDT0000 complete [ 8293.857297] LustreError: 278099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.900701] LustreError: 278099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8296.088502] LustreError: 278849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304939 with bad export cookie 15652340335757807322 [ 8296.089099] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.095697] LustreError: 278849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8296.461040] Lustre: server umount lustre-MDT0001 complete [ 8307.094673] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8310.322480] Key type lgssc unregistered [ 8310.625233] LNet: 281789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8310.634389] LNetError: 281789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8310.677318] LNet: Removed LNI 192.168.202.138@tcp [ 8311.747913] Key type .llcrypt unregistered [ 8311.750399] Key type ._llcrypt unregistered [ 8322.794632] Key type ._llcrypt registered [ 8322.802745] Key type .llcrypt registered [ 8322.958644] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8339.573880] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 8354.938195] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8356.422844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8356.440063] alg: No test for adler32 (adler32-zlib) [ 8357.528703] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 8357.884893] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8359.576614] Key type lgssc registered [ 8360.609308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8394.790716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8394.820783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.085656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8396.109240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8396.171870] Lustre: lustre-MDT0000: new disk, initializing [ 8396.242354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8396.262894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8401.527235] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8411.508330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8411.607422] Lustre: 285335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8411.656408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8411.660172] Lustre: Skipped 1 previous similar message [ 8411.721907] Lustre: lustre-MDT0001: new disk, initializing [ 8411.803816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.854590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8411.871524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8416.021616] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8424.785738] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8430.893753] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8436.854541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.120916] Lustre: lustre-OST0000: new disk, initializing [ 8437.124657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8437.133145] Lustre: 287304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8437.210676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8442.997780] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8444.524626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8444.534981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8444.698437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8452.961790] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8457.977713] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8458.226271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8462.677667] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8462.922514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8465.388305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8465.401099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.408157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8465.416549] Lustre: Skipped 1 previous similar message [ 8470.509697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8470.526416] Lustre: Skipped 2 previous similar messages [ 8475.623359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8475.630546] Lustre: Skipped 1 previous similar message [ 8479.199150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8479.324988] Lustre: server umount lustre-OST0000 complete [ 8483.808051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8483.824581] LustreError: Skipped 1 previous similar message [ 8483.834759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.845338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.860616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.593639] Lustre: server umount lustre-MDT0000 complete [ 8492.310621] LustreError: 285327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305135 with bad export cookie 15220425894437999842 [ 8492.312721] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.326443] LustreError: 285327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8492.627389] Lustre: server umount lustre-MDT0001 complete [ 8499.997626] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:12:22 (1789305142) [ 8507.360347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.853946] LustreError: 289717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.877066] LustreError: 289717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8507.943401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.507661] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8512.992477] LustreError: 289718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8518.115061] LustreError: 289717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.104601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.685492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8526.786977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8534.175437] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8538.994194] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8544.973146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8545.246413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.464813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8556.604330] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8566.753741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8566.765652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.778825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8566.786444] Lustre: Skipped 2 previous similar messages [ 8568.811564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8571.068668] Lustre: server umount lustre-OST0000 complete [ 8578.016596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8578.026273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8578.037795] Lustre: Skipped 1 previous similar message [ 8578.047450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8578.052524] Lustre: Skipped 2 previous similar messages [ 8588.255154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8588.415330] Lustre: server umount lustre-MDT0000 complete [ 8589.282520] LustreError: 289717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.298348] LustreError: 289717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8591.182472] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305234 with bad export cookie 15220425894438000717 [ 8591.183438] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8591.191409] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8591.409395] Lustre: server umount lustre-MDT0001 complete [ 8601.132889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8604.296963] Key type lgssc unregistered [ 8604.644823] LNet: 293253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8604.653539] LNetError: 293253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8605.677482] LNet: Removed LNI 192.168.202.138@tcp [ 8606.507458] Key type .llcrypt unregistered [ 8606.513172] Key type ._llcrypt unregistered [ 8630.673944] Key type ._llcrypt registered [ 8630.678826] Key type .llcrypt registered [ 8630.793968] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 8643.526653] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8644.553490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8644.578159] alg: No test for adler32 (adler32-zlib) [ 8645.607762] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 8645.818744] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8647.471122] Key type lgssc registered [ 8648.731610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8687.275646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8687.316601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.605963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8688.638224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8688.735491] Lustre: lustre-MDT0000: new disk, initializing [ 8688.809931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.838181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8693.317061] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8703.707950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.804959] Lustre: 297055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8703.864700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8703.868475] Lustre: Skipped 1 previous similar message [ 8703.935728] Lustre: lustre-MDT0001: new disk, initializing [ 8704.025374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8704.107191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8704.120876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8707.939377] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8716.052107] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8721.266865] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8726.644990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8726.903581] Lustre: lustre-OST2710: new disk, initializing [ 8726.906811] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8726.911119] Lustre: 299022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8726.976492] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8732.641890] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8734.747865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8734.756079] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8734.868794] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8740.778754] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8746.495261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8746.630509] Lustre: lustre-OST03e8: new disk, initializing [ 8746.633706] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8746.637403] Lustre: 300085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8746.705771] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8752.143763] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8754.787145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8754.800900] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8754.932974] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8758.418134] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8763.802921] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8763.962913] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8768.064662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8768.278514] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8772.301981] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8772.486574] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8776.453681] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8776.686371] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8780.776790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8780.778496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.785203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.819417] Lustre: Skipped 2 previous similar messages [ 8785.891245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.895515] Lustre: Skipped 6 previous similar messages [ 8786.285290] Lustre: server umount lustre-MDT0000 complete [ 8789.453582] LustreError: 297046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305432 with bad export cookie 15693279396645663258 [ 8789.460858] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.462350] LustreError: 297046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8789.720838] Lustre: server umount lustre-MDT0001 complete [ 8803.326986] Lustre: server umount lustre-OST2710 complete [ 8816.474883] Lustre: server umount lustre-OST03e8 complete [ 8830.315228] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 8838.813392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8869.839203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.007377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8870.034247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8870.133202] Lustre: lustre-MDT0000: new disk, initializing [ 8870.184142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.203797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8874.423978] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8884.420552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.567532] Lustre: 305334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.633397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.638396] Lustre: Skipped 1 previous similar message [ 8884.718743] Lustre: lustre-MDT0001: new disk, initializing [ 8884.804602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8884.840348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.849821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8889.093498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8897.422841] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8902.978928] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8908.152821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.348295] Lustre: lustre-OST0000: new disk, initializing [ 8908.351945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8908.357763] Lustre: 307300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8908.427965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8913.541899] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8915.000924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8915.009270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8915.089784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8921.202423] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8924.787182] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8924.954554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8928.418652] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8928.572503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8930.789950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8930.790725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.793664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8930.797489] LustreError: Skipped 1 previous similar message [ 8930.813685] Lustre: Skipped 2 previous similar messages [ 8935.908894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8935.923605] Lustre: Skipped 2 previous similar messages [ 8936.310384] Lustre: server umount lustre-OST0000 complete [ 8941.024767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8941.032933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.048432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8941.057191] Lustre: Skipped 1 previous similar message [ 8945.871721] Lustre: server umount lustre-MDT0000 complete [ 8949.338656] LustreError: 305328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305592 with bad export cookie 15693279396645664938 [ 8949.340601] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8949.348268] LustreError: 305328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8949.711377] Lustre: server umount lustre-MDT0001 complete [ 8957.877542] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:20:00 (1789305600) [ 8959.239637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8960.988651] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:20:03 (1789305603) [ 8964.416752] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8985.988742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.240515] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8986.270477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.369148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8990.304123] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8997.180595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.261944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8997.292941] Lustre: 311159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9000.916951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9020.663154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.879406] Lustre: Denying initial registration attempt from nid 192.168.202.138@tcp, specified as failover [ 9020.886996] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9020.893810] LustreError: 312612:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9020.899505] LustreError: 312612:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9020.903403] LustreError: 312612:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9020.968945] Lustre: server umount lustre-OST0000 complete [ 9020.977132] LustreError: 312612:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9027.552810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.553401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.563981] Lustre: Skipped 1 previous similar message [ 9027.575545] Lustre: Skipped 2 previous similar messages [ 9032.682385] LustreError: 311170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.700678] LustreError: 311170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9032.796568] Lustre: server umount lustre-MDT0000 complete [ 9036.135390] LustreError: 311909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305679 with bad export cookie 15693279396645665820 [ 9036.147055] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9036.159791] LustreError: 311909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9036.458452] Lustre: server umount lustre-MDT0001 complete [ 9045.896539] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9048.250313] Key type lgssc unregistered [ 9048.497391] LNet: 313730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9048.510268] LNetError: 313730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9048.534972] LNet: Removed LNI 192.168.202.138@tcp [ 9049.229119] Key type .llcrypt unregistered [ 9049.230771] Key type ._llcrypt unregistered [ 9058.211213] Key type ._llcrypt registered [ 9058.216925] Key type .llcrypt registered [ 9058.326204] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9059.270055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9059.303120] alg: No test for adler32 (adler32-zlib) [ 9060.313960] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9060.605797] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9062.295193] Key type lgssc registered [ 9063.233782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9103.058191] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9109.989309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9110.020919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.233310] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9111.260299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9111.414414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9115.483305] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9122.563691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.645801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9122.688328] Lustre: 316470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9122.959550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9126.635928] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9133.042432] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.834642] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9157.929314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.105721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9158.241216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.482136] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9169.677181] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9174.503764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9174.515210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.533501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9174.536065] Lustre: Skipped 1 previous similar message [ 9177.473459] Lustre: server umount lustre-OST0000 complete [ 9184.748499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9184.758133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.759782] LustreError: Skipped 1 previous similar message [ 9184.760621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9184.789699] Lustre: Skipped 2 previous similar messages [ 9186.561591] Lustre: server umount lustre-MDT0000 complete [ 9189.345684] LustreError: 316462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305832 with bad export cookie 8200846927782665553 [ 9189.348194] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9189.359892] LustreError: 316462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9189.612281] Lustre: server umount lustre-MDT0001 complete [ 9197.487809] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9199.507537] Key type lgssc unregistered [ 9199.716867] LNet: 320044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9199.720206] LNetError: 320044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9199.735514] LNet: Removed LNI 192.168.202.138@tcp [ 9200.403117] Key type .llcrypt unregistered [ 9200.405831] Key type ._llcrypt unregistered [ 9220.164207] Key type ._llcrypt registered [ 9220.165515] Key type .llcrypt registered [ 9220.257981] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9221.291783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9221.456586] alg: No test for adler32 (adler32-zlib) [ 9222.495683] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9222.719305] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9224.384425] Key type lgssc registered [ 9225.431761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9231.773460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9231.790309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.307498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9237.154428] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9243.621432] LustreError: 321437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.629527] LustreError: 321437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9244.106365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.427392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.070089] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9254.045882] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9258.182531] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9265.637313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9265.974981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.707358] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9276.754171] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9288.633712] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 9288.672556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.695582] Lustre: Skipped 2 previous similar messages [ 9290.084900] Lustre: server umount lustre-MDT0000 complete [ 9292.991447] LustreError: 321418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305936 with bad export cookie 12298078016940994918 [ 9293.011668] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9293.721689] LustreError: 321432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.732116] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 9293.755920] Lustre: Skipped 3 previous similar messages [ 9293.792504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9298.847687] LustreError: 321432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.848942] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 9298.867348] LustreError: 321432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9298.871696] Lustre: Skipped 1 previous similar message [ 9299.328302] Lustre: server umount lustre-MDT0001 complete [ 9304.034463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9310.118712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.371509] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaaab8ccc63cf0d43 [ 9318.382822] Lustre: MGC192.168.202.138@tcp: Connection restored to 0@lo (at 0@lo) [ 9318.617153] LustreError: 323265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.638143] LustreError: 323265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9318.698180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9318.757889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9321.721588] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9323.482517] LustreError: 325031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.916323] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9328.241675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.492269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9331.572384] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9333.740390] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9336.086935] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9337.045930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9340.818293] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9341.869548] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9349.087944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9349.098416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.108333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.119101] Lustre: Skipped 1 previous similar message [ 9349.121995] Lustre: Skipped 1 previous similar message [ 9354.221027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9354.229664] Lustre: Skipped 2 previous similar messages [ 9359.329089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9359.479819] Lustre: server umount lustre-OST0000 complete [ 9364.450569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9364.452840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.488915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.495024] Lustre: Skipped 3 previous similar messages [ 9368.267967] Lustre: server umount lustre-MDT0000 complete [ 9369.572203] LustreError: 325031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.583853] LustreError: 325031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9370.964585] LustreError: 321417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306014 with bad export cookie 12298078016941002051 [ 9370.976122] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9370.980874] LustreError: 321417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9371.205338] Lustre: server umount lustre-MDT0001 complete [ 9377.984796] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9379.610406] Key type lgssc unregistered [ 9379.834439] LNet: 327700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9379.838912] LNetError: 327700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9379.847258] LNet: Removed LNI 192.168.202.138@tcp [ 9380.323648] Key type .llcrypt unregistered [ 9380.326135] Key type ._llcrypt unregistered [ 9397.490622] Key type ._llcrypt registered [ 9397.492990] Key type .llcrypt registered [ 9397.573157] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9398.398694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9398.436120] alg: No test for adler32 (adler32-zlib) [ 9399.392428] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9399.549734] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9401.183167] Key type lgssc registered [ 9401.993810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9407.479250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9407.502668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.792889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9411.882032] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9418.212592] LustreError: 329091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9418.235158] LustreError: 329091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9418.334703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.578104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.328087] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9428.042456] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9432.473770] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9438.176217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.177585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9438.189920] Lustre: Skipped 1 previous similar message [ 9441.164810] Lustre: server umount lustre-MDT0000 complete [ 9443.298651] LustreError: 329091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.311300] LustreError: 329091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9443.702474] LustreError: 330706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306086 with bad export cookie 419947827725271037 [ 9443.709043] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9443.709258] LustreError: 330706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9443.910446] Lustre: server umount lustre-MDT0001 complete [ 9454.080638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9454.307644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9454.323867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9454.464884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9457.349502] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9463.326164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.398703] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9463.425831] Lustre: 331677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9463.601281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9466.432597] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9471.192879] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9474.746905] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9481.835164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9482.001769] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9482.008430] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9482.013628] LustreError: 333558:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9482.016649] LustreError: 333558:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9482.023293] LustreError: 333558:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9482.071351] Lustre: server umount lustre-OST0000 complete [ 9482.074737] LustreError: 333558:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9490.023696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9490.204459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9490.334274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.331944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9499.081540] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.745402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9504.168357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.232051] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9504.291760] Lustre: lustre-OST0001: new disk, initializing [ 9504.294777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9504.300159] Lustre: 335100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9504.360713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.292961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9506.298056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9506.357488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9508.233944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9513.049583] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9516.519277] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.523261] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9516.525663] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9516.525671] Lustre: Skipped 1 previous similar message [ 9516.533195] Lustre: Skipped 1 previous similar message [ 9516.536144] LustreError: Skipped 1 previous similar message [ 9521.633079] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9521.643405] Lustre: Skipped 2 previous similar messages [ 9526.754386] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9526.759326] Lustre: Skipped 1 previous similar message [ 9529.311136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9529.434562] Lustre: server umount lustre-OST0001 complete [ 9531.875166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.877880] LustreError: 334026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.878137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9531.886389] Lustre: Skipped 1 previous similar message [ 9531.905704] LustreError: 334026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9536.998463] LustreError: 334732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9537.005031] LustreError: 334732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9542.115698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.118211] LustreError: 334026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9542.119236] Lustre: Skipped 4 previous similar messages [ 9546.207748] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9546.274576] Lustre: server umount lustre-OST0000 complete [ 9550.818243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9550.821851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.828556] Lustre: server umount lustre-MDT0000 complete [ 9557.296955] LustreError: 331669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306200 with bad export cookie 419947827725271590 [ 9557.300504] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.305836] LustreError: 331669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9557.542258] Lustre: server umount lustre-MDT0001 complete [ 9563.555913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9565.262485] Key type lgssc unregistered [ 9565.500042] LNet: 337057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9565.507690] LNetError: 337057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9565.517985] LNet: Removed LNI 192.168.202.138@tcp [ 9566.081597] Key type .llcrypt unregistered [ 9566.083614] Key type ._llcrypt unregistered [ 9616.058466] Key type ._llcrypt registered [ 9616.061720] Key type .llcrypt registered [ 9616.152501] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 9625.540436] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9626.106113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9626.115987] alg: No test for adler32 (adler32-zlib) [ 9627.102946] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9627.337032] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9628.951316] Key type lgssc registered [ 9629.933888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9652.851259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9652.876374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.129039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9654.152863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9654.213051] Lustre: lustre-MDT0000: new disk, initializing [ 9654.275589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.310499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9657.107398] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9664.785644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.850890] Lustre: 341960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9664.879122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9664.882228] Lustre: Skipped 1 previous similar message [ 9664.932307] Lustre: lustre-MDT0001: new disk, initializing [ 9664.999353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.023534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9665.032464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9668.170389] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9676.030189] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9680.232379] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9685.337775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9685.646265] Lustre: lustre-OST0000: new disk, initializing [ 9685.651786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9685.661068] Lustre: 343928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9685.724150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9690.299374] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9695.774324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9695.786789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9695.856557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9696.835487] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9700.610422] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9700.798106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9704.325030] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9704.473779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9705.957515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9705.959392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9705.969652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9705.977832] LustreError: Skipped 1 previous similar message [ 9705.982888] Lustre: Skipped 1 previous similar message [ 9711.078597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9711.085542] Lustre: Skipped 1 previous similar message [ 9716.198233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9716.205941] Lustre: Skipped 1 previous similar message [ 9720.287156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9720.421917] Lustre: server umount lustre-OST0000 complete [ 9726.434351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.436880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9726.449688] Lustre: Skipped 2 previous similar messages [ 9726.457540] Lustre: Skipped 1 previous similar message [ 9729.677522] Lustre: server umount lustre-MDT0000 complete [ 9732.389548] LustreError: 341952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306375 with bad export cookie 1723693010796527689 [ 9732.392307] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9732.397038] LustreError: 341952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9732.624491] Lustre: server umount lustre-MDT0001 complete [ 9738.141995] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:33:00 (1789306380) [ 9742.949578] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:33:05 (1789306385) [ 9748.391467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.649686] LustreError: 346681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.660750] LustreError: 346681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9748.686636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.459578] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9754.086363] LustreError: 346682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.161855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.379209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9760.624193] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9765.914915] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9769.564402] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9773.817927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9774.033201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.985103] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9784.579533] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9801.279943] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9802.526395] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9803.620612] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9804.640942] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9805.857230] Lustre: Failing over lustre-MDT0000 [ 9806.047523] Lustre: server umount lustre-MDT0000 complete [ 9806.760453] LustreError: 348519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.867292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.868264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9808.869049] LustreError: 347408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.880805] Lustre: Skipped 2 previous similar messages [ 9813.991578] LustreError: 346681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.999332] LustreError: 346681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9821.235086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.300740] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9821.443612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9821.478020] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9822.103115] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9824.038571] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9826.801321] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9826.823529] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9826.856571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9830.458226] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9831.432801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9834.046720] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9835.464698] Lustre: Failing over lustre-MDT0000 [ 9835.616591] Lustre: server umount lustre-MDT0000 complete [ 9837.027903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.030086] LustreError: 348519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.039908] Lustre: Skipped 2 previous similar messages [ 9837.049034] LustreError: 348519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9851.853433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.928852] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9852.095732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9852.832589] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9854.753670] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9857.526442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9857.532339] Lustre: Skipped 2 previous similar messages [ 9857.539465] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9857.563979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9861.071852] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9862.240686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9867.746256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.746797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9867.747243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.760254] Lustre: Skipped 1 previous similar message [ 9872.698293] Lustre: server umount lustre-MDT0000 complete [ 9872.864773] LustreError: 346682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.880595] LustreError: 346682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9874.937395] LustreError: 346662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306518 with bad export cookie 1723693010796530398 [ 9874.940740] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9874.942421] LustreError: 346662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9875.111603] Lustre: server umount lustre-MDT0001 complete [ 9887.728291] Lustre: server umount lustre-OST0000 complete [ 9898.729550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.100254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.104216] Lustre: Skipped 1 previous similar message [ 9902.266744] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9908.638705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.137668] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9917.135996] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9920.841848] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9925.591711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9930.342743] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9936.248689] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9945.329069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9950.691728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.696153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9950.704660] Lustre: Skipped 2 previous similar messages [ 9950.718406] Lustre: Skipped 3 previous similar messages [ 9951.791134] Lustre: server umount lustre-OST0000 complete [ 9954.784354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9960.379533] Lustre: server umount lustre-MDT0000 complete [ 9960.930460] LustreError: 353035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9960.948497] LustreError: 353035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9962.883130] LustreError: 353015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306606 with bad export cookie 1723693010796531147 [ 9962.886518] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9962.900139] LustreError: 353015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9963.107474] Lustre: server umount lustre-MDT0001 complete [ 9969.620163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9971.133531] Key type lgssc unregistered [ 9971.303661] LNet: 356581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9971.313442] LNetError: 356581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9971.332815] LNet: Removed LNI 192.168.202.138@tcp [ 9971.793549] Key type .llcrypt unregistered [ 9971.795579] Key type ._llcrypt unregistered [ 9985.901196] Key type ._llcrypt registered [ 9985.902392] Key type .llcrypt registered [ 9985.957442] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9986.637535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9986.662652] alg: No test for adler32 (adler32-zlib) [ 9987.612657] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [ 9987.725609] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9989.328143] Key type lgssc registered [ 9990.114868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9994.470802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9994.481562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.692930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.740664] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10002.944662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.154148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10005.345477] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10008.325055] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10011.070705] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10013.990480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10014.165278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10017.303591] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10020.333165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10021.103790] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10038.753912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10038.757907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10038.763435] Lustre: Skipped 2 previous similar messages [10038.770698] Lustre: Skipped 2 previous similar messages [10043.208546] Lustre: server umount lustre-MDT0000 complete [10043.873265] LustreError: 360418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.880149] LustreError: 360418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10045.214636] LustreError: 358051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306688 with bad export cookie 5260470125784671274 [10045.216969] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.220355] LustreError: 358051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10045.352405] Lustre: server umount lustre-MDT0001 complete [10057.610860] Lustre: server umount lustre-OST0000 complete [10064.755610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.011431] LustreError: 361794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.054503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10067.408112] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10070.496371] LustreError: 361795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10072.542602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.757948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10075.652806] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10080.051308] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10083.289572] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10086.805611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10087.001708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.091880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10090.286887] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10094.738980] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10103.776170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10103.782206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.791353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.874601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.875190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.883149] Lustre: Skipped 1 previous similar message [10107.907017] Lustre: Skipped 1 previous similar message [10109.179744] Lustre: server umount lustre-MDT0000 complete [10111.507259] LustreError: 361774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306754 with bad export cookie 5260470125784672373 [10111.510480] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10111.514713] LustreError: 361774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10111.773541] Lustre: server umount lustre-MDT0001 complete [10124.399230] Lustre: server umount lustre-OST0000 complete [10134.184075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10134.446370] LustreError: 365702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10134.490356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10137.187184] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10139.621675] LustreError: 365703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10142.374410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.104976] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10149.720584] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10153.495695] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10157.950197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10159.216607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10161.956194] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10166.571167] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10169.825210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10169.829165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.838800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10169.842448] Lustre: Skipped 1 previous similar message [10172.901341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [10175.190261] Lustre: server umount lustre-OST0000 complete [10178.017570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10178.020078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.030216] Lustre: Skipped 2 previous similar messages [10178.035976] Lustre: Skipped 3 previous similar messages [10183.476259] Lustre: server umount lustre-MDT0000 complete [10185.433323] LustreError: 365684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306828 with bad export cookie 5260470125784673395 [10185.439156] LustreError: 366431:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [10185.441898] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.447351] LustreError: 366431:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [10185.665378] Lustre: server umount lustre-MDT0001 complete [10191.028646] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10192.554560] Key type lgssc unregistered [10192.735876] LNet: 369229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10192.739779] LNetError: 369229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10192.756940] LNet: Removed LNI 192.168.202.138@tcp [10193.271435] Key type .llcrypt unregistered [10193.272863] Key type ._llcrypt unregistered [10207.635716] Key type ._llcrypt registered [10207.637587] Key type .llcrypt registered [10207.691291] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10208.334915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10208.344593] alg: No test for adler32 (adler32-zlib) [10209.281788] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10209.416739] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10211.039221] Key type lgssc registered [10211.657972] Lustre: Echo OBD driver; http://www.lustre.org/ [10215.698527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10215.708548] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.712574] LustreError: 370644:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10215.721459] LustreError: 370644:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10215.724551] LustreError: 370644:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10215.727209] LustreError: 370644:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10215.730959] LustreError: 370644:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10219.426622] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10219.429571] LustreError: 371040:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10219.439862] LustreError: 371040:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10219.443741] LustreError: 371040:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10219.447867] LustreError: 371040:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10219.452089] LustreError: 371040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10228.336048] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10229.656461] Key type lgssc unregistered [10229.811330] LNet: 372041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10229.815496] LNetError: 372041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10229.828822] LNet: Removed LNI 192.168.202.138@tcp [10230.228329] Key type .llcrypt unregistered [10230.230338] Key type ._llcrypt unregistered [10238.197234] Key type ._llcrypt registered [10238.198302] Key type .llcrypt registered [10238.245612] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [10246.028792] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10246.676294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10246.710749] alg: No test for adler32 (adler32-zlib) [10247.650597] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10247.789621] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10249.399134] Key type lgssc registered [10250.014138] Lustre: Echo OBD driver; http://www.lustre.org/ [10267.569966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10267.588116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10268.726820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10268.755683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10268.807245] Lustre: lustre-MDT0000: new disk, initializing [10268.856737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10268.864859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10270.785793] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10275.817505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.865407] Lustre: 375166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10275.892365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10275.895081] Lustre: Skipped 1 previous similar message [10275.932187] Lustre: lustre-MDT0001: new disk, initializing [10275.975162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10275.992297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10275.996335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10277.712920] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10282.667354] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10285.376385] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10288.705431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.839686] Lustre: lustre-OST0000: new disk, initializing [10288.842242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10288.846163] Lustre: 377135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10288.891437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10291.842878] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10295.301964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10295.307924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10295.380278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10297.330899] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10299.566751] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10299.675652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10301.741308] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10301.841380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10305.505942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10305.508421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.509431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10305.513432] Lustre: Skipped 1 previous similar message [10310.633178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10310.636235] Lustre: Skipped 2 previous similar messages [10315.749425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10317.280052] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10317.412070] Lustre: server umount lustre-OST0000 complete [10320.865342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.872788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.877626] Lustre: Skipped 1 previous similar message [10320.886126] Lustre: Skipped 2 previous similar messages [10325.376354] Lustre: server umount lustre-MDT0000 complete [10327.183774] LustreError: 375158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306970 with bad export cookie 14932252539931505186 [10327.185268] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10327.193240] LustreError: 375158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10327.373091] Lustre: server umount lustre-MDT0001 complete [10331.481767] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10332.406513] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:42:55 (1789306975) [10335.736085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.988046] LustreError: 379589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.995662] LustreError: 379589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10336.032678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.196225] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10341.345973] LustreError: 379589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10341.358690] LustreError: 379589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10342.389862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10342.589399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10344.704012] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10348.437985] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10351.389959] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10354.914260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10355.074921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10357.881711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10361.535216] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10365.008366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10365.089636] Lustre: lustre-OST0001: new disk, initializing [10365.092752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10365.095687] Lustre: 382422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10368.460375] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10373.658105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10373.662882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10373.704544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10373.998454] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10378.726385] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.727448] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10378.740259] LustreError: Skipped 2 previous similar messages [10378.745266] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10378.747494] Lustre: Skipped 1 previous similar message [10382.876551] Lustre: server umount lustre-OST0001 complete [10383.846676] LustreError: 383099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.857368] LustreError: 383099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10388.960568] LustreError: 381409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.974546] LustreError: 381409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10394.082246] LustreError: 383099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.091142] LustreError: 383099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10397.154420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10397.158224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.165446] Lustre: Skipped 1 previous similar message [10397.168801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10397.171065] Lustre: Skipped 1 previous similar message [10402.328403] Lustre: server umount lustre-OST0000 complete [10409.439859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10409.441334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.443514] LustreError: Skipped 1 previous similar message [10409.450164] Lustre: Skipped 1 previous similar message [10410.594877] Lustre: server umount lustre-MDT0000 complete [10412.372943] LustreError: 379570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307055 with bad export cookie 14932252539931506054 [10412.375223] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10412.381098] LustreError: 379570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10418.728564] Lustre: server umount lustre-MDT0001 complete [10424.205562] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10425.795853] Key type lgssc unregistered [10425.958446] LNet: 384404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10425.963824] LNetError: 384404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10425.975968] LNet: Removed LNI 192.168.202.138@tcp [10426.373504] Key type .llcrypt unregistered [10426.375267] Key type ._llcrypt unregistered [10448.539408] Key type ._llcrypt registered [10448.540855] Key type .llcrypt registered [10448.601660] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10449.005114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10449.047819] alg: No test for adler32 (adler32-zlib) [10449.944918] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10450.072209] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10451.679154] Key type lgssc registered [10452.127410] Lustre: Echo OBD driver; http://www.lustre.org/ [10455.410880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10455.419963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.569669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10456.583147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10456.663842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10458.919703] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10463.404142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.475099] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10463.491403] Lustre: 387176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10463.698346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10466.093751] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10470.124969] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10473.216445] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.536779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.539353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10477.546328] Lustre: Skipped 1 previous similar message [10480.427179] Lustre: server umount lustre-MDT0000 complete [10482.116354] LustreError: 387166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307125 with bad export cookie 4576841624361571448 [10482.119157] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10482.121498] LustreError: 387166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10482.261613] Lustre: server umount lustre-MDT0001 complete [10486.468348] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:45:29 (1789307129) [10489.468876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.820069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10491.663179] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10494.945842] LustreError: 389686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.958965] LustreError: 389686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10495.720183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.909265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10497.998986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10500.968355] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10503.742981] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10506.784691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.886812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10506.978538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10509.577544] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10512.921640] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10517.264608] Lustre: Permanently deactivating lustre-OST0000 [10517.272970] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10519.357719] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10521.086230] LustreError: 392571:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10521.090156] LustreError: 392571:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10521.092995] LustreError: 392571:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10522.074543] LustreError: 392619:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10522.077980] LustreError: 392619:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10528.993494] Lustre: server umount lustre-OST0000 complete [10530.200484] LustreError: 392868:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10530.203436] LustreError: 392868:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10531.810682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.810981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10531.817320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10531.822313] Lustre: Skipped 2 previous similar messages [10536.928583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10536.931214] Lustre: Skipped 1 previous similar message [10541.060736] Lustre: server umount lustre-MDT0000 complete [10542.048079] LustreError: 390411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10542.054346] LustreError: 390411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10542.576094] LustreError: 391516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307185 with bad export cookie 4576841624361572029 [10542.578371] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10542.585595] LustreError: 391516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10552.872087] Lustre: server umount lustre-MDT0001 complete [10557.056395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.855634] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10559.950623] LustreError: 394025:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10562.597829] Lustre: 394363:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.138@tcp [10563.501723] Lustre: server umount MGS complete [10568.381927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.626211] LustreError: 395086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.636948] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10568.639954] Lustre: Skipped 1 previous similar message [10568.674349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10570.646295] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10573.795279] LustreError: 395087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.526512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.701449] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10576.529687] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10580.004940] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10582.752375] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10583.683637] Lustre: Permanently reactivating lustre-OST0000 [10583.689794] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10583.691862] Lustre: Skipped 2 previous similar messages [10610.143805] LustreError: 396911:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10614.305724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10614.484182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10614.488173] Lustre: Skipped 1 previous similar message [10617.201667] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10620.596726] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10630.111818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10630.114687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.118596] Lustre: Skipped 1 previous similar message [10630.120701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10631.137455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10633.500556] Lustre: server umount lustre-OST0000 complete [10636.259671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.259717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10636.262020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10636.262026] Lustre: Skipped 2 previous similar messages [10636.269449] Lustre: Skipped 2 previous similar messages [10641.642348] Lustre: server umount lustre-MDT0000 complete [10643.318306] LustreError: 395067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307286 with bad export cookie 4576841624361573079 [10643.320533] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10643.329744] LustreError: 395067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10643.498968] Lustre: server umount lustre-MDT0001 complete [10648.294510] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10649.630713] Key type lgssc unregistered [10649.790295] LNet: 399686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10649.793780] LNetError: 399686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10649.803337] LNet: Removed LNI 192.168.202.138@tcp [10650.212309] Key type .llcrypt unregistered [10650.213922] Key type ._llcrypt unregistered [10657.953367] Key type ._llcrypt registered [10657.954822] Key type .llcrypt registered [10658.018662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [10665.019495] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10665.387357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.425852] alg: No test for adler32 (adler32-zlib) [10666.300374] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10666.400119] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10667.993800] Key type lgssc registered [10668.439335] Lustre: Echo OBD driver; http://www.lustre.org/ [10681.906157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10681.915195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.011632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10683.025691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10683.084751] Lustre: lustre-MDT0000: new disk, initializing [10683.130241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10683.138147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10684.889367] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10690.014310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.066845] Lustre: 402811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10690.096553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10690.099568] Lustre: Skipped 1 previous similar message [10690.136606] Lustre: lustre-MDT0001: new disk, initializing [10690.175462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10690.189320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10690.193678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10691.937570] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10696.440042] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10698.913995] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10702.017642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10702.170261] Lustre: lustre-OST0000: new disk, initializing [10702.172748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10702.176908] Lustre: 404777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10702.232093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10705.160941] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10707.983625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10707.995073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10708.061802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10710.394989] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10712.118739] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10712.203061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10714.148376] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10714.233139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10718.179171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.179506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10718.184575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10718.188150] LustreError: Skipped 1 previous similar message [10718.192325] Lustre: Skipped 1 previous similar message [10721.360412] Lustre: server umount lustre-OST0000 complete [10723.296585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.297281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10723.301120] Lustre: Skipped 2 previous similar messages [10728.416291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10728.418775] Lustre: Skipped 2 previous similar messages [10729.315480] Lustre: server umount lustre-MDT0000 complete [10731.009723] LustreError: 402847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307374 with bad export cookie 15145233242233969539 [10731.013088] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10731.016614] LustreError: 402847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10731.154322] Lustre: server umount lustre-MDT0001 complete [10734.801873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10735.544681] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:49:38 (1789307378) [10738.462636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.694833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10740.488472] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10743.776205] LustreError: 407235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.781983] LustreError: 407235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10744.309182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.482764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10746.279955] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10749.245309] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10751.412257] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10754.253976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.399387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10756.956831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10759.804453] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10760.537132] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10766.327396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10768.864778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10768.866049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.868104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10768.873942] Lustre: Skipped 1 previous similar message [10773.974574] Lustre: server umount lustre-OST0000 complete [10775.523142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.526903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.532556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.536402] Lustre: Skipped 1 previous similar message [10781.597560] Lustre: server umount lustre-MDT0000 complete [10782.984665] LustreError: 409074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307426 with bad export cookie 15145233242233970421 [10782.985954] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10782.989659] LustreError: 409074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10783.107628] Lustre: server umount lustre-MDT0001 complete [10787.435340] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10788.680690] Key type lgssc unregistered [10788.830906] LNet: 410956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10788.834146] LNetError: 410956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10788.849592] LNet: Removed LNI 192.168.202.138@tcp [10789.230954] Key type .llcrypt unregistered [10789.232428] Key type ._llcrypt unregistered [10799.696973] Key type ._llcrypt registered [10799.698917] Key type .llcrypt registered [10799.751422] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10799.918524] Key type .llcrypt unregistered [10799.920428] Key type ._llcrypt unregistered [10808.707376] Key type ._llcrypt registered [10808.708953] Key type .llcrypt registered [10808.755578] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10809.090257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10809.122056] alg: No test for adler32 (adler32-zlib) [10809.988303] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10810.083311] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10811.671134] Key type lgssc registered [10812.146967] Lustre: Echo OBD driver; http://www.lustre.org/ [10815.285173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10815.291951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.469772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10818.038408] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10821.531893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10821.658730] LustreError: 413133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.664819] LustreError: 413133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10821.667903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10823.966343] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10826.721993] LustreError: 413136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.727782] LustreError: 413136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10826.846841] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10827.747892] LustreError: 413133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10827.753113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10827.754087] LustreError: 413133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10829.669807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.850476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10831.514275] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10840.031948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10840.035825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.041211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10845.157512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10845.163056] Lustre: Skipped 2 previous similar messages [10845.456177] Lustre: server umount lustre-OST0000 complete [10849.761320] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10849.763945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.767868] Lustre: Skipped 1 previous similar message [10849.769834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10852.951477] Lustre: server umount lustre-MDT0000 complete [10854.321366] LustreError: 414875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307497 with bad export cookie 875557255889255809 [10854.323653] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10854.326355] LustreError: 414875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10854.441787] Lustre: server umount lustre-MDT0001 complete [10858.619723] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10859.769085] Key type lgssc unregistered [10859.938505] LNet: 416278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10859.942260] LNetError: 416278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10859.951562] LNet: Removed LNI 192.168.202.138@tcp [10860.327123] Key type .llcrypt unregistered [10860.328164] Key type ._llcrypt unregistered [10870.954432] Key type ._llcrypt registered [10870.955773] Key type .llcrypt registered [10871.000028] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10871.359390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10871.407913] alg: No test for adler32 (adler32-zlib) [10872.281655] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10872.383798] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10873.983198] Key type lgssc registered [10874.456713] Lustre: Echo OBD driver; http://www.lustre.org/ [10877.591126] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10877.598983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10893.023433] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10898.143275] LustreError: 417640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [10924.767211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10926.547708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10928.739825] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10930.887274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.030686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10931.051444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10932.251081] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10934.790785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.905305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10936.217271] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10938.539827] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10940.371404] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10944.481464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10944.483294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10944.486882] Lustre: Skipped 1 previous similar message [10949.601454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10949.603601] Lustre: Skipped 2 previous similar messages [10949.963852] Lustre: server umount lustre-OST0000 complete [10954.720680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10954.722141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10954.725657] Lustre: Skipped 1 previous similar message [10957.270236] Lustre: server umount lustre-MDT0000 complete [10958.560094] LustreError: 417646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307601 with bad export cookie 5305063580891465467 [10958.563377] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10958.565369] LustreError: 417646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10958.672966] Lustre: server umount lustre-MDT0001 complete [10962.491133] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10963.597519] Key type lgssc unregistered [10963.733371] LNet: 421208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10963.737112] LNetError: 421208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10963.743299] LNet: Removed LNI 192.168.202.138@tcp [10964.138851] Key type .llcrypt unregistered [10964.140271] Key type ._llcrypt unregistered [10973.255455] Key type ._llcrypt registered [10973.256698] Key type .llcrypt registered [10973.291901] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10973.603092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10973.608898] alg: No test for adler32 (adler32-zlib) [10974.464813] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [10974.563378] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10976.151216] Key type lgssc registered [10976.529838] Lustre: Echo OBD driver; http://www.lustre.org/ [10979.131355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10979.136652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.271205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10981.580504] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10984.267453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.374961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10985.638946] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10987.940311] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.688098] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10991.819591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10991.926283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10993.666919] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10995.842582] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11005.223466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11010.015899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11010.019740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.026100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.529781] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.530506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.534542] Lustre: Skipped 1 previous similar message [11010.542203] Lustre: Skipped 1 previous similar message [11012.510512] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [11014.417151] Lustre: server umount lustre-MDT0000 complete [11015.207283] LustreError: 422588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.38@tcp arrived at 1789307658 with bad export cookie 1822027336415224168 [11015.212847] LustreError: 422588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11015.648534] LustreError: 423348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.653634] LustreError: 423348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11015.868215] LustreError: 422589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307659 with bad export cookie 1822027336415223517 [11015.872149] LustreError: 422589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11015.872458] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11019.743752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11019.746335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.750395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11020.767902] LustreError: 423348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.772882] LustreError: 423348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11021.964231] Lustre: server umount lustre-OST0000 complete [11024.259805] Lustre: server umount lustre-MDT0001 complete [11027.924500] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11028.972127] Key type lgssc unregistered [11029.098404] LNet: 426310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11029.101242] LNetError: 426310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11029.111304] LNet: Removed LNI 192.168.202.138@tcp [11029.431545] Key type .llcrypt unregistered [11029.432577] Key type ._llcrypt unregistered [11038.675822] Key type ._llcrypt registered [11038.676707] Key type .llcrypt registered [11038.712282] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11039.024101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11039.061046] alg: No test for adler32 (adler32-zlib) [11039.913871] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [11040.001214] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11041.575236] Key type lgssc registered [11041.924733] Lustre: Echo OBD driver; http://www.lustre.org/ [11044.361322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11044.367960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.499171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11046.834090] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11049.586204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.711814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11051.063471] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11053.424405] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11055.294909] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11057.446062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11057.554605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11059.407410] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11061.838511] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11070.883956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11075.551837] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11075.554570] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.558693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11076.066277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.066477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11076.073267] Lustre: Skipped 1 previous similar message [11077.532157] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [11079.445748] Lustre: server umount lustre-MDT0001 complete [11081.184165] LustreError: 427701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.184707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11081.184871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.185552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11081.190440] LustreError: 427701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11086.304044] LustreError: 428439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.304581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11087.118083] Lustre: server umount lustre-OST0000 complete [11088.444957] Lustre: server umount lustre-MDT0000 complete [11092.876924] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11093.957435] Key type lgssc unregistered [11094.093376] LNet: 431402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11094.095814] LNetError: 431402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11094.108290] LNet: Removed LNI 192.168.202.138@tcp [11094.411752] Key type .llcrypt unregistered [11094.412859] Key type ._llcrypt unregistered [11102.367340] Key type ._llcrypt registered [11102.368741] Key type .llcrypt registered [11102.406989] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [11108.643230] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11108.968315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11108.981064] alg: No test for adler32 (adler32-zlib) [11109.839393] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [11109.932945] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11111.519159] Key type lgssc registered [11111.961861] Lustre: Echo OBD driver; http://www.lustre.org/ [11124.291732] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11129.451860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11129.459107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.543308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11130.555174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11130.594545] Lustre: lustre-MDT0000: new disk, initializing [11130.631196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11130.638656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11132.240990] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11138.080720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11138.180366] Lustre: lustre-OST0000: new disk, initializing [11138.182797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11138.184378] Lustre: Skipped 1 previous similar message [11138.186148] Lustre: 436402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11138.210976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11138.300540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11138.303703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11138.318743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11140.166857] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11143.852033] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11149.216582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11149.264342] Lustre: 435386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11149.293215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11149.327840] Lustre: lustre-MDT0001: new disk, initializing [11149.366292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11149.380357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11149.385577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11150.742414] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11154.240018] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11154.307599] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11155.631546] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11156.719800] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [11161.059028] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11161.059113] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.059849] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11161.070280] Lustre: Skipped 1 previous similar message [11162.012318] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [11162.014437] Lustre: Skipped 1 previous similar message [11164.302766] Lustre: server umount lustre-MDT0001 complete [11171.297617] LustreError: 435391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.303530] LustreError: 435391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11176.417013] LustreError: 438087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.423129] LustreError: 438087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11179.487763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11179.490983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11179.497501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.536567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.536990] LustreError: 435395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.738090] Lustre: server umount lustre-OST0000 complete [11190.324537] Lustre: server umount lustre-MDT0000 complete [11194.871884] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11195.963519] Key type lgssc unregistered [11196.098598] LNet: 440150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11196.101293] LNetError: 440150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11196.111663] LNet: Removed LNI 192.168.202.138@tcp [11196.429909] Key type .llcrypt unregistered [11196.433753] Key type ._llcrypt unregistered [11210.058985] Key type ._llcrypt registered [11210.060166] Key type .llcrypt registered [11210.100836] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11210.207432] Key type .llcrypt unregistered [11210.208840] Key type ._llcrypt unregistered [11217.070318] Key type ._llcrypt registered [11217.071763] Key type .llcrypt registered [11217.115275] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11217.448450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11217.458476] alg: No test for adler32 (adler32-zlib) [11218.329496] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [11218.436482] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11220.023163] Key type lgssc registered [11220.404881] Lustre: Echo OBD driver; http://www.lustre.org/ [11229.555437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11229.561155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.651684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11230.661381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11230.698610] Lustre: lustre-MDT0000: new disk, initializing [11230.719276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11230.726264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11232.208331] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11236.568320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.612556] Lustre: 443487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11236.632695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11236.634345] Lustre: Skipped 1 previous similar message [11236.666554] Lustre: lustre-MDT0001: new disk, initializing [11236.700871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11236.716242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11236.719996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11238.098462] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11242.125890] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11244.132532] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11246.459321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11246.580478] Lustre: lustre-OST0000: new disk, initializing [11246.582655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11246.586128] Lustre: 445454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11246.620568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11248.725812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11252.225947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11252.232305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11252.269184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11252.667374] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11257.313399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11257.314859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11257.315797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11257.323016] Lustre: Skipped 2 previous similar messages [11261.216263] Lustre: server umount lustre-MDT0000 complete [11262.511230] LustreError: 443479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307905 with bad export cookie 6055286640314255947 [11262.513463] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.516501] LustreError: 443479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11262.632345] Lustre: server umount lustre-MDT0001 complete [11274.202430] Lustre: server umount lustre-OST0000 complete [11277.746677] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:58:40 (1789307920) [11280.294164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11280.448217] LustreError: 447477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.454335] LustreError: 447477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11280.472065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11281.846690] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11284.696190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.825726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11286.163712] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11288.672761] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11290.574978] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11306.083027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11306.150447] Lustre: Found index 0 for lustre-OST0000, updating log [11306.211448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11308.304104] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11310.664938] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11326.694043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11335.135715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11335.138076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.142381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11335.144009] Lustre: Skipped 2 previous similar messages [11336.163282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11336.165582] Lustre: Skipped 2 previous similar messages [11338.768027] Lustre: server umount lustre-OST0000 complete [11351.519635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11351.522031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11351.524678] Lustre: Skipped 1 previous similar message [11351.526071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11357.328887] Lustre: server umount lustre-MDT0000 complete [11357.664918] LustreError: 448206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.669607] LustreError: 448206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11358.459079] LustreError: 447458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308001 with bad export cookie 6055286640314257340 [11358.461997] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11358.463790] LustreError: 447458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11358.561255] Lustre: server umount lustre-MDT0001 complete [11361.038055] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:00:04 (1789308004) [11367.967290] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [11941.223727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.364306] LustreError: 453909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11941.387611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.641200] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11945.272589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.378140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11946.605162] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11948.757551] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11950.326664] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11965.474372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.552557] Lustre: Found index 0 for lustre-OST0000, updating log [11965.653748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11967.530095] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11968.742866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11969.979582] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11971.682097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.685844] Lustre: Skipped 1 previous similar message [11971.687731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11971.689431] Lustre: Skipped 2 previous similar messages [11972.706535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11973.729872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11976.783478] Lustre: server umount lustre-OST0000 complete [11978.102511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11978.107123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11978.109904] Lustre: Skipped 2 previous similar messages [11982.240389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11982.243343] Lustre: Skipped 7 previous similar messages [11984.282087] Lustre: server umount lustre-MDT0000 complete [11984.289622] LustreError: 453928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.295522] LustreError: 453928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11985.509187] LustreError: 453890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308628 with bad export cookie 6055286640314258411 [11985.511597] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11985.513510] LustreError: 453890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11985.624771] Lustre: server umount lustre-MDT0001 complete [11994.655314] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [12002.662500] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12005.373101] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12016.365903] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:10:59 (1789308659) [12022.045115] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12023.094909] Key type lgssc unregistered [12028.319351] LNet: 460543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12028.322312] LNetError: 460543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12096.933400] LNet: Removed LNI 192.168.202.138@tcp [12097.261678] Key type .llcrypt unregistered [12097.262856] Key type ._llcrypt unregistered [12103.753449] Key type ._llcrypt registered [12103.754731] Key type .llcrypt registered [12103.791782] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12104.083282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12104.109081] alg: No test for adler32 (adler32-zlib) [12104.942379] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12105.024519] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12106.607161] Key type lgssc registered [12106.932509] Lustre: Echo OBD driver; http://www.lustre.org/ [12108.231301] LNet: Added LNI 192.168.202.138@tcp100 [8/256/0/180] [12108.507145] LNet: Added LNI 192.168.202.138@tcp101 [8/256/0/180] [12108.777064] LNet: Added LNI 192.168.202.138@tcp102 [8/256/0/180] [12109.042758] LNet: Added LNI 192.168.202.138@tcp103 [8/256/0/180] [12109.300308] LNet: Added LNI 192.168.202.138@tcp104 [8/256/0/180] [12109.551723] LNet: Added LNI 192.168.202.138@tcp105 [8/256/0/180] [12109.813606] LNet: Added LNI 192.168.202.138@tcp106 [8/256/0/180] [12110.078382] LNet: Added LNI 192.168.202.138@tcp107 [8/256/0/180] [12110.345669] LNet: Added LNI 192.168.202.138@tcp108 [8/256/0/180] [12110.601697] LNet: Added LNI 192.168.202.138@tcp109 [8/256/0/180] [12110.855822] LNet: Added LNI 192.168.202.138@tcp110 [8/256/0/180] [12111.115563] LNet: Added LNI 192.168.202.138@tcp111 [8/256/0/180] [12111.378255] LNet: Added LNI 192.168.202.138@tcp112 [8/256/0/180] [12111.633468] LNet: Added LNI 192.168.202.138@tcp113 [8/256/0/180] [12111.887862] LNet: Added LNI 192.168.202.138@tcp114 [8/256/0/180] [12112.159076] LNet: Added LNI 192.168.202.138@tcp115 [8/256/0/180] [12112.424564] LNet: Added LNI 192.168.202.138@tcp116 [8/256/0/180] [12112.679798] LNet: Added LNI 192.168.202.138@tcp117 [8/256/0/180] [12112.941720] LNet: Added LNI 192.168.202.138@tcp118 [8/256/0/180] [12113.206299] LNet: Added LNI 192.168.202.138@tcp119 [8/256/0/180] [12113.476202] LNet: Added LNI 192.168.202.138@tcp120 [8/256/0/180] [12113.741519] LNet: Added LNI 192.168.202.138@tcp121 [8/256/0/180] [12114.011306] LNet: Added LNI 192.168.202.138@tcp122 [8/256/0/180] [12114.272874] LNet: Added LNI 192.168.202.138@tcp123 [8/256/0/180] [12114.539739] LNet: Added LNI 192.168.202.138@tcp124 [8/256/0/180] [12114.797647] LNet: Added LNI 192.168.202.138@tcp125 [8/256/0/180] [12115.046522] LNet: Added LNI 192.168.202.138@tcp126 [8/256/0/180] [12115.316656] LNet: Added LNI 192.168.202.138@tcp127 [8/256/0/180] [12115.606992] LNet: Added LNI 192.168.202.138@tcp128 [8/256/0/180] [12115.903779] LNet: Added LNI 192.168.202.138@tcp129 [8/256/0/180] [12116.193950] LNet: Added LNI 192.168.202.138@tcp130 [8/256/0/180] [12116.492641] LNet: Added LNI 192.168.202.138@tcp131 [8/256/0/180] [12116.781937] LNet: Added LNI 192.168.202.138@tcp132 [8/256/0/180] [12117.080420] LNet: Added LNI 192.168.202.138@tcp133 [8/256/0/180] [12117.368923] LNet: Added LNI 192.168.202.138@tcp134 [8/256/0/180] [12117.672090] LNet: Added LNI 192.168.202.138@tcp135 [8/256/0/180] [12119.859973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12119.865112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.902569] Lustre: 463537:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [12120.932086] LNetError: 460998:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12120.946182] LNetError: Protocol error connecting to 192.168.202.138@tcp121 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12120.949968] LNetError: Skipped 3 previous similar messages [12120.995839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12121.005788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12121.046970] Lustre: lustre-MDT0000: new disk, initializing [12121.067877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12121.073243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12121.439189] LNet: 4 local NIs in recovery (showing 4): 192.168.202.138@tcp128, 192.168.202.138@tcp127, 192.168.202.138@tcp129, 192.168.202.138@tcp121 [12121.441626] LNetError: 461000:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12121.447812] LNetError: 461000:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [12121.454757] LNetError: Protocol error connecting to 192.168.202.138@tcp110 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12121.458086] LNetError: Skipped 5 previous similar messages [12122.397180] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12123.488396] LNetError: 460998:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12123.493710] LNetError: 460998:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [12123.497686] LNetError: Protocol error connecting to 192.168.202.138@tcp127 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12123.502783] LNetError: Skipped 1 previous similar message [12126.476027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12126.523369] Lustre: 463552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12126.547030] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12126.549031] Lustre: Skipped 1 previous similar message [12126.574848] Lustre: 464359:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [12126.577297] Lustre: 464359:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12126.585620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12126.590741] Lustre: lustre-MDT0001: new disk, initializing [12126.639544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12126.650954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12126.655210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12127.584482] LNetError: 461001:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12127.589136] LNetError: Protocol error connecting to 192.168.202.138@tcp127 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12127.926750] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12131.502236] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12133.245620] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12135.424722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12135.537834] Lustre: lustre-OST0000: new disk, initializing [12135.540331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12135.544468] Lustre: 465529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12135.579628] Lustre: 465534:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [12135.582726] Lustre: 465534:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [12135.599385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12135.776358] LNetError: 463562:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12135.779656] LNetError: Protocol error connecting to 192.168.202.138@tcp127 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12137.522498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12140.023776] Lustre: 466089:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [12140.027673] Lustre: 466089:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12140.056446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12140.062049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12140.122110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12141.363961] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12145.121407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12145.123105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.123766] LustreError: Skipped 1 previous similar message [12145.123791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12145.129166] Lustre: Skipped 1 previous similar message [12149.969561] Lustre: server umount lustre-OST0000 complete [12151.136272] LNetError: 463558:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [12151.139902] LNetError: Protocol error connecting to 192.168.202.138@tcp127 at host 192.168.202.138:988: is it running a compatible version of Lustre? [12152.287800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12152.291382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.297395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12152.299949] Lustre: Skipped 1 previous similar message [12155.360150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.360749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.350145] Lustre: server umount lustre-MDT0000 complete [12158.554513] LustreError: 463547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308801 with bad export cookie 12710634960173380521 [12158.557245] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.557789] LustreError: 463547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12158.665571] Lustre: server umount lustre-MDT0001 complete [12161.984973] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12162.944711] Key type lgssc unregistered [12163.093699] LNet: 467437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12163.096424] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12163.098620] LNet: Removed LNI 192.168.202.138@tcp [12164.127165] LNet: Removed LNI 192.168.202.138@tcp100 [12164.128620] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12164.131600] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12164.134923] LNet: Removed LNI 192.168.202.138@tcp101 [12165.151150] LNet: Removed LNI 192.168.202.138@tcp102 [12165.152475] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12165.155304] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12165.157599] LNet: Removed LNI 192.168.202.138@tcp103 [12166.175116] LNet: Removed LNI 192.168.202.138@tcp104 [12167.199103] LNet: Removed LNI 192.168.202.138@tcp105 [12167.200416] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12167.202451] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12167.204615] LNet: Removed LNI 192.168.202.138@tcp106 [12168.223134] LNet: Removed LNI 192.168.202.138@tcp107 [12169.247158] LNet: Removed LNI 192.168.202.138@tcp108 [12170.271130] LNet: Removed LNI 192.168.202.138@tcp109 [12171.295184] LNet: Removed LNI 192.168.202.138@tcp110 [12171.296763] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12171.299183] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12171.301745] LNet: Removed LNI 192.168.202.138@tcp111 [12172.319162] LNet: Removed LNI 192.168.202.138@tcp112 [12173.343158] LNet: Removed LNI 192.168.202.138@tcp113 [12174.367181] LNet: Removed LNI 192.168.202.138@tcp114 [12175.391150] LNet: Removed LNI 192.168.202.138@tcp115 [12176.415155] LNet: Removed LNI 192.168.202.138@tcp116 [12177.439131] LNet: Removed LNI 192.168.202.138@tcp117 [12178.463157] LNet: Removed LNI 192.168.202.138@tcp118 [12179.487151] LNet: Removed LNI 192.168.202.138@tcp119 [12179.488544] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12179.491758] LNetError: 467437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12179.494721] LNet: Removed LNI 192.168.202.138@tcp120 [12180.511139] LNet: Removed LNI 192.168.202.138@tcp121 [12181.535145] LNet: Removed LNI 192.168.202.138@tcp122 [12182.559155] LNet: Removed LNI 192.168.202.138@tcp123 [12183.583154] LNet: Removed LNI 192.168.202.138@tcp124 [12184.607092] LNet: Removed LNI 192.168.202.138@tcp125 [12185.631176] LNet: Removed LNI 192.168.202.138@tcp126 [12185.633514] LNet: Removed LNI 192.168.202.138@tcp127 [12186.655121] LNet: Removed LNI 192.168.202.138@tcp128 [12187.679138] LNet: Removed LNI 192.168.202.138@tcp129 [12187.681107] LNet: Removed LNI 192.168.202.138@tcp130 [12187.682173] LNet: Removed LNI 192.168.202.138@tcp131 [12187.684083] LNet: Removed LNI 192.168.202.138@tcp132 [12187.685930] LNet: Removed LNI 192.168.202.138@tcp133 [12187.688031] LNet: Removed LNI 192.168.202.138@tcp134 [12187.695550] LNet: Removed LNI 192.168.202.138@tcp135 [12187.973480] Key type .llcrypt unregistered [12187.974607] Key type ._llcrypt unregistered [12195.175725] Key type ._llcrypt registered [12195.176759] Key type .llcrypt registered [12195.211543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12201.570432] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12201.971473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12202.014833] alg: No test for adler32 (adler32-zlib) [12202.869282] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12202.972879] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12204.559103] Key type lgssc registered [12204.927084] Lustre: Echo OBD driver; http://www.lustre.org/ [12215.790707] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:14:18 (1789308858) [12221.572911] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12222.565551] Key type lgssc unregistered [12222.703431] LNet: 471489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12222.705670] LNetError: 471489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12222.716298] LNet: Removed LNI 192.168.202.138@tcp [12222.999431] Key type .llcrypt unregistered [12223.001080] Key type ._llcrypt unregistered [12229.445018] Key type ._llcrypt registered [12229.445983] Key type .llcrypt registered [12229.479837] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12229.812448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12229.842051] alg: No test for adler32 (adler32-zlib) [12230.696467] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12230.791218] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12232.383123] Key type lgssc registered [12232.728176] Lustre: Echo OBD driver; http://www.lustre.org/ [12235.743278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12235.748389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.828182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12236.839565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12236.875848] Lustre: lustre-MDT0000: new disk, initializing [12236.896169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12236.903249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12238.208170] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12242.229033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.269205] Lustre: 472676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12242.291045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12242.292851] Lustre: Skipped 1 previous similar message [12242.326862] Lustre: lustre-MDT0001: new disk, initializing [12242.357693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12242.368390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12242.371730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12243.753169] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12247.404506] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12249.142931] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12251.272312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12251.383402] Lustre: lustre-OST0000: new disk, initializing [12251.385651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12251.389110] Lustre: 474641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12251.421259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12253.292096] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12256.868647] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12258.039792] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lnet_if_list [12259.684634] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12259.696733] LNet: Added LNI 192.168.202.138@tcp10 [8/256/0/180] [12260.869200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12260.872606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12260.924572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12261.645849] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12261.655932] LNet: Added LNI 192.168.202.138@tcp11 [8/256/0/180] [12263.024331] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12263.033819] LNet: Added LNI 192.168.202.138@tcp12 [8/256/0/180] [12264.282725] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12264.291902] LNet: Added LNI 192.168.202.138@tcp13 [8/256/0/180] [12265.609494] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12265.618872] LNet: Added LNI 192.168.202.138@tcp14 [8/256/0/180] [12267.088991] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12267.104661] LNet: Added LNI 192.168.202.138@tcp15 [8/256/0/180] [12268.579164] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12268.591392] LNet: Added LNI 192.168.202.138@tcp16 [8/256/0/180] [12270.133249] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12270.144679] LNet: Added LNI 192.168.202.138@tcp17 [8/256/0/180] [12271.754625] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12271.770114] LNet: Added LNI 192.168.202.138@tcp18 [8/256/0/180] [12273.290629] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12273.300629] LNet: Added LNI 192.168.202.138@tcp19 [8/256/0/180] [12274.853619] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12274.863638] LNet: Added LNI 192.168.202.138@tcp20 [8/256/0/180] [12276.427271] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12276.438117] LNet: Added LNI 192.168.202.138@tcp21 [8/256/0/180] [12278.089827] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12278.098923] LNet: Added LNI 192.168.202.138@tcp22 [8/256/0/180] [12279.824592] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12279.834128] LNet: Added LNI 192.168.202.138@tcp23 [8/256/0/180] [12281.458317] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12281.469515] LNet: Added LNI 192.168.202.138@tcp24 [8/256/0/180] [12283.438559] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12283.449759] LNet: Added LNI 192.168.202.138@tcp25 [8/256/0/180] [12285.458227] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12285.472491] LNet: Added LNI 192.168.202.138@tcp26 [8/256/0/180] [12287.244866] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12287.263874] LNet: Added LNI 192.168.202.138@tcp27 [8/256/0/180] [12289.562364] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12289.574100] LNet: Added LNI 192.168.202.138@tcp28 [8/256/0/180] [12291.490626] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12291.499406] LNet: Added LNI 192.168.202.138@tcp29 [8/256/0/180] [12293.320415] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12293.331406] LNet: Added LNI 192.168.202.138@tcp30 [8/256/0/180] [12295.260425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12295.269938] LNet: Added LNI 192.168.202.138@tcp31 [8/256/0/180] [12297.097594] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12297.108288] LNet: Added LNI 192.168.202.138@tcp32 [8/256/0/180] [12299.016198] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12299.028516] LNet: Added LNI 192.168.202.138@tcp33 [8/256/0/180] [12301.149996] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12301.159630] LNet: Added LNI 192.168.202.138@tcp34 [8/256/0/180] [12303.195438] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12303.205845] LNet: Added LNI 192.168.202.138@tcp35 [8/256/0/180] [12305.434059] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12305.445201] LNet: Added LNI 192.168.202.138@tcp36 [8/256/0/180] [12307.549879] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12307.559832] LNet: Added LNI 192.168.202.138@tcp37 [8/256/0/180] [12309.788759] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12309.804417] LNet: Added LNI 192.168.202.138@tcp38 [8/256/0/180] [12312.146829] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12312.156047] LNet: Added LNI 192.168.202.138@tcp39 [8/256/0/180] [12314.315875] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12314.326079] LNet: Added LNI 192.168.202.138@tcp40 [8/256/0/180] [12316.634486] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12316.645201] LNet: Added LNI 192.168.202.138@tcp41 [8/256/0/180] [12318.906305] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12318.920400] LNet: Added LNI 192.168.202.138@tcp42 [8/256/0/180] [12321.330334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12321.342971] LNet: Added LNI 192.168.202.138@tcp43 [8/256/0/180] [12323.594707] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12323.604800] LNet: Added LNI 192.168.202.138@tcp44 [8/256/0/180] [12326.069586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12326.079773] LNet: Added LNI 192.168.202.138@tcp45 [8/256/0/180] [12328.400218] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12328.410227] LNet: Added LNI 192.168.202.138@tcp46 [8/256/0/180] [12330.890377] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12330.900686] LNet: Added LNI 192.168.202.138@tcp47 [8/256/0/180] [12333.570980] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12333.583000] LNet: Added LNI 192.168.202.138@tcp48 [8/256/0/180] [12336.264172] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12336.273604] LNet: Added LNI 192.168.202.138@tcp49 [8/256/0/180] [12338.855184] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12338.865076] LNet: Added LNI 192.168.202.138@tcp50 [8/256/0/180] [12347.873615] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12347.874731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.877013] Lustre: Skipped 2 previous similar messages [12347.880147] Lustre: Skipped 2 previous similar messages [12352.159669] Lustre: server umount lustre-MDT0000 complete [12353.407192] LustreError: 472666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308996 with bad export cookie 1049248514325087330 [12353.410455] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12353.411020] LustreError: 472666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12353.519439] Lustre: server umount lustre-MDT0001 complete [12360.452763] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12363.290240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12374.495134] Lustre: 471947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789309001/real 1789309001] req@000000005424101c x1876226340894592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789309017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12374.503986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.591465] LustreError: 471945:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000830948cf x1876226340895360/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12378.689538] LustreError: 474634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12378.697408] LustreError: 474634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12378.704335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12379.743121] Lustre: 471947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789309006/real 1789309006] req@000000005858eb34 x1876226340894720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789309022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12379.750722] LustreError: 474635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12379.978510] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12382.851549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.994761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12384.040258] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12384.473826] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12385.055148] Lustre: 471948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789309011/real 1789309011] req@00000000c9c00654 x1876226340894976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789309027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12386.921399] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12388.749548] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12389.343144] Lustre: 471949:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789309016/real 1789309016] req@000000005e6a992e x1876226340895104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789309032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12398.560388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12398.562637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.566671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12399.072144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12399.586738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12399.589683] Lustre: Skipped 1 previous similar message [12402.515928] Lustre: server umount lustre-OST0000 complete [12404.705112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.705746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12404.710662] Lustre: Skipped 2 previous similar messages [12404.713812] Lustre: Skipped 1 previous similar message [12409.824115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12409.826123] Lustre: Skipped 1 previous similar message [12418.015116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12418.079596] Lustre: server umount lustre-MDT0000 complete [12419.340122] LustreError: 472667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309062 with bad export cookie 1049248514325089325 [12419.342923] LustreError: 472667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12419.343121] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.448479] Lustre: server umount lustre-MDT0001 complete [12423.109405] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12424.179396] Key type lgssc unregistered [12424.328354] LNet: 501601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12424.331739] LNetError: 501601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12424.334378] LNet: Removed LNI 192.168.202.138@tcp [12424.336242] LNet: Removed LNI 192.168.202.138@tcp10 [12424.337513] LNet: Removed LNI 192.168.202.138@tcp11 [12424.338941] LNet: Removed LNI 192.168.202.138@tcp12 [12424.340264] LNet: Removed LNI 192.168.202.138@tcp13 [12424.341980] LNet: Removed LNI 192.168.202.138@tcp14 [12424.343362] LNet: Removed LNI 192.168.202.138@tcp15 [12424.344500] LNet: Removed LNI 192.168.202.138@tcp16 [12424.345931] LNet: Removed LNI 192.168.202.138@tcp17 [12424.347069] LNet: Removed LNI 192.168.202.138@tcp18 [12424.348219] LNet: Removed LNI 192.168.202.138@tcp19 [12424.349333] LNet: Removed LNI 192.168.202.138@tcp20 [12424.350994] LNet: Removed LNI 192.168.202.138@tcp21 [12424.353279] LNet: Removed LNI 192.168.202.138@tcp22 [12424.355393] LNet: Removed LNI 192.168.202.138@tcp23 [12424.357360] LNet: Removed LNI 192.168.202.138@tcp24 [12424.358463] LNet: Removed LNI 192.168.202.138@tcp25 [12424.359563] LNet: Removed LNI 192.168.202.138@tcp26 [12424.361379] LNet: Removed LNI 192.168.202.138@tcp27 [12424.363632] LNet: Removed LNI 192.168.202.138@tcp28 [12424.365720] LNet: Removed LNI 192.168.202.138@tcp29 [12424.367967] LNet: Removed LNI 192.168.202.138@tcp30 [12424.370151] LNet: Removed LNI 192.168.202.138@tcp31 [12424.372392] LNet: Removed LNI 192.168.202.138@tcp32 [12424.374679] LNet: Removed LNI 192.168.202.138@tcp33 [12424.376922] LNet: Removed LNI 192.168.202.138@tcp34 [12424.379061] LNet: Removed LNI 192.168.202.138@tcp35 [12424.381123] LNet: Removed LNI 192.168.202.138@tcp36 [12424.383349] LNet: Removed LNI 192.168.202.138@tcp37 [12424.385607] LNet: Removed LNI 192.168.202.138@tcp38 [12424.387700] LNet: Removed LNI 192.168.202.138@tcp39 [12424.389894] LNet: Removed LNI 192.168.202.138@tcp40 [12424.392048] LNet: Removed LNI 192.168.202.138@tcp41 [12424.394259] LNet: Removed LNI 192.168.202.138@tcp42 [12424.396357] LNet: Removed LNI 192.168.202.138@tcp43 [12424.398183] LNet: Removed LNI 192.168.202.138@tcp44 [12424.400184] LNet: Removed LNI 192.168.202.138@tcp45 [12424.402273] LNet: Removed LNI 192.168.202.138@tcp46 [12424.404431] LNet: Removed LNI 192.168.202.138@tcp47 [12424.406599] LNet: Removed LNI 192.168.202.138@tcp48 [12424.408616] LNet: Removed LNI 192.168.202.138@tcp49 [12424.416325] LNet: Removed LNI 192.168.202.138@tcp50 [12424.672078] Key type .llcrypt unregistered [12424.673197] Key type ._llcrypt unregistered [12431.082115] Key type ._llcrypt registered [12431.083065] Key type .llcrypt registered [12431.127446] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12437.226214] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12437.673225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12437.682429] alg: No test for adler32 (adler32-zlib) [12438.532122] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12438.617946] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12440.199159] Key type lgssc registered [12440.573697] Lustre: Echo OBD driver; http://www.lustre.org/ [12451.533070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12451.538735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.614326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12452.623246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12452.659534] Lustre: lustre-MDT0000: new disk, initializing [12452.679507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12452.685554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12454.022220] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12458.152242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12458.194908] Lustre: 504765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12458.217653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12458.219447] Lustre: Skipped 1 previous similar message [12458.253377] Lustre: lustre-MDT0001: new disk, initializing [12458.285886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12458.297962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12458.301495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12459.602572] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12463.250138] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12465.010864] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12467.774202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12467.867208] Lustre: *** cfs_fail_loc=90f, val=0*** [12467.897570] Lustre: lustre-OST0000: new disk, initializing [12467.899558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12467.903107] Lustre: 506782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12467.929894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12469.772789] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12473.334199] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12473.844606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12473.848225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12473.885982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12478.947025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12478.947325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.950501] LustreError: Skipped 1 previous similar message [12478.951138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12478.955947] Lustre: Skipped 1 previous similar message [12481.804837] Lustre: server umount lustre-OST0000 complete [12484.065230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.065349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12484.068954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12484.073753] Lustre: Skipped 2 previous similar messages [12489.184738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12489.186810] Lustre: Skipped 1 previous similar message [12494.304528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.306176] Lustre: Skipped 1 previous similar message [12497.375131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12497.440124] Lustre: server umount lustre-MDT0000 complete [12498.593593] LustreError: 505562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309141 with bad export cookie 802061964158619308 [12498.597516] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12498.598708] LustreError: 505562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12498.710935] Lustre: server umount lustre-MDT0001 complete [12502.150439] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12503.088275] Key type lgssc unregistered [12503.211354] LNet: 508686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12503.214158] LNetError: 508686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12503.221234] LNet: Removed LNI 192.168.202.138@tcp [12503.534351] Key type .llcrypt unregistered [12503.535195] Key type ._llcrypt unregistered [12513.088890] Key type ._llcrypt registered [12513.090607] Key type .llcrypt registered [12513.156139] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12513.643718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12513.683927] alg: No test for adler32 (adler32-zlib) [12514.600854] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12514.723421] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12516.327172] Key type lgssc registered [12516.896329] Lustre: Echo OBD driver; http://www.lustre.org/ [12521.858657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12521.899407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12523.515132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12528.111701] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12533.728600] LustreError: 509931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.742419] LustreError: 509931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12535.913990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.339828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12540.524891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12547.855514] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12551.774982] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12555.310711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12555.482464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12558.980151] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12563.856473] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12576.142827] loop0: detected capacity change from 0 to 204800000 [12576.224321] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12576.229849] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12576.239551] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12578.853508] loop0: detected capacity change from 0 to 204800000 [12578.970923] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12578.975465] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12578.981436] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12581.130832] loop0: detected capacity change from 0 to 204800000 [12583.987520] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12584.098584] Lustre: fs15246-OST0000: new disk, initializing [12584.101293] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12584.105123] Lustre: 513522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12584.140885] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12586.973891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12594.320840] loop1: detected capacity change from 0 to 204800000 [12599.446518] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12599.536402] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12599.585159] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12599.665171] Lustre: fs15246-MDT0000: new disk, initializing [12599.919244] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12599.959110] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12600.195782] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12601.248260] Lustre: 514643:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006208b252 x1876226638649600/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789309260 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12603.857872] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12610.342580] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12610.365404] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12610.402937] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12628.457918] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.466990] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12631.251670] Lustre: server umount fs15246-MDT0000 complete [12633.573937] LustreError: 511771:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.697927] LustreError: 511771:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.845989] LustreError: 509927:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.889371] Lustre: server umount fs15246-OST0000 complete [12666.389168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12666.394032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12666.412155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12669.415773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12669.420213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12669.424601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12669.449601] Lustre: Skipped 1 previous similar message [12671.399704] Lustre: server umount lustre-OST0000 complete [12679.649789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.667259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12679.689136] Lustre: Skipped 1 previous similar message [12679.693937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12684.771904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12684.780805] Lustre: Skipped 2 previous similar messages [12690.399263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12690.786427] Lustre: server umount lustre-MDT0000 complete [12694.017170] LustreError: 510658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.350141] LustreError: 509910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309342 with bad export cookie 2745846104906756411 [12699.353880] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12699.376137] LustreError: 509910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12699.796527] Lustre: server umount lustre-MDT0001 complete [12715.342867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12718.820517] Key type lgssc unregistered [12719.256669] LNet: 517743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12719.266249] LNetError: 517743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12719.280432] LNet: Removed LNI 192.168.202.138@tcp [12720.503407] Key type .llcrypt unregistered [12720.505550] Key type ._llcrypt unregistered [12740.699433] Key type ._llcrypt registered [12740.701886] Key type .llcrypt registered [12740.789211] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12742.072315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12742.081918] alg: No test for adler32 (adler32-zlib) [12743.134570] Lustre: Lustre: Build Version: 2.17.57_108_gfed9771 [12743.487401] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12745.265160] Key type lgssc registered [12746.497783] Lustre: Echo OBD driver; http://www.lustre.org/ [12785.860523] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12796.930306] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12835.980477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12836.000170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.209736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12837.237295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12837.358751] Lustre: lustre-MDT0000: new disk, initializing [12837.477118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12837.516702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12842.660661] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12853.582669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12853.762450] Lustre: 522207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12853.850339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12853.857715] Lustre: Skipped 1 previous similar message [12853.981379] Lustre: lustre-MDT0001: new disk, initializing [12854.095353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.158943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12854.166921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12860.087438] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12870.092779] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12876.218933] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12883.687459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12883.949919] Lustre: lustre-OST0000: new disk, initializing [12883.953586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12883.962207] Lustre: 524177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12884.095313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12891.508611] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12893.802683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12893.820619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12893.993979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12902.703200] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12907.805936] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12908.034388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12913.309282] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12913.556896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12919.777106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.785126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12919.789530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.799686] Lustre: Skipped 1 previous similar message [12922.203313] Lustre: server umount lustre-OST0000 complete [12930.016460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12930.028046] Lustre: Skipped 1 previous similar message [12930.043342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12930.058336] Lustre: Skipped 1 previous similar message [12935.139502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12935.142541] Lustre: Skipped 1 previous similar message [12940.258797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12940.276821] Lustre: Skipped 1 previous similar message [12941.279686] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12941.601734] Lustre: server umount lustre-MDT0000 complete [12946.203197] LustreError: 522199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309589 with bad export cookie 16565044043282468670 [12946.211276] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12946.223302] LustreError: 522199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12946.670483] Lustre: server umount lustre-MDT0001 complete [12977.003811] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12990.913940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13033.185555] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12740 sec ========= 10:27:54 (1789309674) [13035.104969] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:27:56 (1789309676) === [13040.797739] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:28:01 (1789309681) === [13088.784617] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13092.358949] Key type lgssc unregistered [13092.807730] LNet: 530827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13092.814819] LNetError: 530827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13092.837461] LNet: Removed LNI 192.168.202.138@tcp [13093.904497] Key type .llcrypt unregistered [13093.906745] Key type ._llcrypt unregistered