[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 579881883 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003265] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007779] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.022450] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027517] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.032213] *** VALIDATE proc *** [ 0.033217] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035223] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.042700] debug: unmapping init [mem 0xffffffff91459000-0xffffffff91460fff] [ 0.044988] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045804] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047019] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053365] rcu: Hierarchical SRCU implementation. [ 0.055627] smp: Bringing up secondary CPUs ... [ 0.056686] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.060378] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143414] node 0 deferred pages initialised in 77ms [ 0.147150] devtmpfs: initialized [ 0.148216] x86/mm: Memory block size: 128MB [ 0.150444] gcov: version magic: 0x41383552 [ 0.152157] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154231] pinctrl core: initialized pinctrl subsystem [ 0.155190] [ 0.155543] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157009] ** ** [ 0.158012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159012] ** ** [ 0.160009] ** This means that this kernel is built to expose internal ** [ 0.161009] ** IOMMU data structures, which may compromise security on ** [ 0.162012] ** your system. ** [ 0.163009] ** ** [ 0.164012] ** If you see this message and you are not debugging the ** [ 0.165011] ** kernel, report this immediately to your vendor! ** [ 0.166010] ** ** [ 0.167012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168008] ************************************************************* [ 0.169644] NET: Registered protocol family 16 [ 0.170347] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176351] cpuidle: using governor menu [ 0.177869] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179313] PCI: Using configuration type 1 for base access [ 0.181096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192168] cryptd: max_cpu_qlen set to 1000 [ 0.197272] ACPI: Added _OSI(Module Device) [ 0.199017] ACPI: Added _OSI(Processor Device) [ 0.200013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207120] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213414] ACPI: Interpreter enabled [ 0.214048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215013] ACPI: Using IOAPIC for interrupt routing [ 0.217114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219354] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226589] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236191] acpiphp: Slot [2] registered [ 0.237086] acpiphp: Slot [5] registered [ 0.240190] acpiphp: Slot [6] registered [ 0.241123] acpiphp: Slot [7] registered [ 0.243124] acpiphp: Slot [8] registered [ 0.245129] acpiphp: Slot [9] registered [ 0.246123] acpiphp: Slot [10] registered [ 0.248120] acpiphp: Slot [3] registered [ 0.249119] acpiphp: Slot [4] registered [ 0.251161] acpiphp: Slot [11] registered [ 0.253117] acpiphp: Slot [12] registered [ 0.254125] acpiphp: Slot [13] registered [ 0.256113] acpiphp: Slot [14] registered [ 0.258117] acpiphp: Slot [15] registered [ 0.259111] acpiphp: Slot [16] registered [ 0.261128] acpiphp: Slot [17] registered [ 0.263099] acpiphp: Slot [18] registered [ 0.264106] acpiphp: Slot [19] registered [ 0.266102] acpiphp: Slot [20] registered [ 0.267113] acpiphp: Slot [21] registered [ 0.269187] acpiphp: Slot [22] registered [ 0.271100] acpiphp: Slot [23] registered [ 0.272149] acpiphp: Slot [24] registered [ 0.274124] acpiphp: Slot [25] registered [ 0.275130] acpiphp: Slot [26] registered [ 0.277169] acpiphp: Slot [27] registered [ 0.279114] acpiphp: Slot [28] registered [ 0.280109] acpiphp: Slot [29] registered [ 0.282131] acpiphp: Slot [30] registered [ 0.284117] acpiphp: Slot [31] registered [ 0.285119] PCI host bridge to bus 0000:00 [ 0.287025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304933] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308377] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.326028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.330025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.332524] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.336886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.339053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343665] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.368014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384290] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.465020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.513022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.520022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.547011] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580325] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.591019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.601017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.609754] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.614016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641712] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.651016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.659019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.671307] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.674386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680529] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.688100] iommu: Default domain type: Passthrough [ 0.690560] SCSI subsystem initialized [ 0.692098] ACPI: bus type USB registered [ 0.693059] usbcore: registered new interface driver usbfs [ 0.695078] usbcore: registered new interface driver hub [ 0.696059] usbcore: registered new device driver usb [ 0.698170] pps_core: LinuxPPS API ver. 1 registered [ 0.699007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702058] PTP clock support registered [ 0.704088] EDAC MC: Ver: 3.0.0 [ 0.705094] PCI: Using ACPI for IRQ routing [ 0.707596] NetLabel: Initializing [ 0.708007] NetLabel: domain hash size = 128 [ 0.710008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711078] NetLabel: unlabeled traffic allowed by default [ 0.712127] vgaarb: loaded [ 0.713227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.715009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721000] clocksource: Switched to clocksource kvm-clock [ 0.828167] VFS: Disk quotas dquot_6.6.0 [ 0.829855] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.832342] *** VALIDATE ramfs *** [ 0.833336] *** VALIDATE hugetlbfs *** [ 0.835086] pnp: PnP ACPI init [ 0.837549] pnp: PnP ACPI: found 6 devices [ 0.856409] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858438] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.859666] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.861369] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.862763] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.865020] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.867895] NET: Registered protocol family 2 [ 0.869853] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.875099] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.878780] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.884289] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887938] TCP: Hash tables configured (established 65536 bind 65536) [ 0.891066] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.894600] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897785] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.900915] NET: Registered protocol family 1 [ 0.904473] RPC: Registered named UNIX socket transport module. [ 0.906797] RPC: Registered udp transport module. [ 0.908385] RPC: Registered tcp transport module. [ 0.909578] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911198] NET: Registered protocol family 44 [ 0.912488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914639] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916051] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.917503] PCI: CLS 0 bytes, default 64 [ 0.918643] Unpacking initramfs... [ 2.334847] debug: unmapping init [mem 0xffff8ccf3cc54000-0xffff8ccf3ffbffff] [ 2.338995] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.341566] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.344714] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.845186] Initialise system trusted keyrings [ 2.846843] Key type blacklist registered [ 2.848572] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.856799] zbud: loaded [ 2.859622] *** VALIDATE nfs *** [ 2.860823] *** VALIDATE nfs4 *** [ 2.862448] pstore: using deflate compression [ 2.865883] Platform Keyring initialized [ 2.946894] NET: Registered protocol family 38 [ 2.948826] Key type asymmetric registered [ 2.950061] Asymmetric key parser 'x509' registered [ 2.951357] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.953595] io scheduler mq-deadline registered [ 2.954742] io scheduler kyber registered [ 2.955893] io scheduler bfq registered [ 2.957262] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.959426] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.961677] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.964262] ACPI: Power Button [PWRF] [ 2.970302] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.977876] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.993391] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.003089] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.023126] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.051819] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.081105] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.086301] Non-volatile memory driver v1.3 [ 3.087477] Linux agpgart interface v0.103 [ 3.119525] virtio_blk virtio1: [vda] 146136 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.122669] vda: detected capacity change from 0 to 74821632 [ 3.135289] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.138129] vdb: detected capacity change from 0 to 1073741824 [ 3.153847] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.157275] vdc: detected capacity change from 0 to 2621440000 [ 3.172884] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.174823] vdd: detected capacity change from 0 to 2621440000 [ 3.185682] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.189233] vde: detected capacity change from 0 to 4294967296 [ 3.206188] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.208436] vdf: detected capacity change from 0 to 4294967296 [ 3.216209] libphy: Fixed MDIO Bus: probed [ 3.221948] usbcore: registered new interface driver usbserial_generic [ 3.223991] usbserial: USB Serial support registered for generic [ 3.225781] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.230355] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.231762] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.233866] mousedev: PS/2 mouse device common for all mice [ 3.237177] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.237796] rtc_cmos 00:05: RTC can wake from S4 [ 3.244839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.246318] rtc_cmos 00:05: registered as rtc0 [ 3.250886] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.251766] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.251817] intel_pstate: CPU model not supported [ 3.255128] hid: raw HID events driver (C) Jiri Kosina [ 3.264637] usbcore: registered new interface driver usbhid [ 3.266988] usbhid: USB HID core driver [ 3.268885] drop_monitor: Initializing network drop monitor service [ 3.271746] Initializing XFRM netlink socket [ 3.274222] NET: Registered protocol family 10 [ 3.277514] Segment Routing with IPv6 [ 3.279305] NET: Registered protocol family 17 [ 3.281732] mpls_gso: MPLS GSO support [ 3.287584] RAS: Correctable Errors collector initialized. [ 3.290030] AVX version of gcm_enc/dec engaged. [ 3.291798] AES CTR mode by8 optimization enabled [ 3.372748] sched_clock: Marking stable (3372715188, 0)->(4283037561, -910322373) [ 3.376340] registered taskstats version 1 [ 3.377930] Loading compiled-in X.509 certificates [ 3.379615] zswap: loaded using pool lzo/zbud [ 3.404699] Key type big_key registered [ 3.417363] Key type encrypted registered [ 3.419424] ima: No TPM chip found, activating TPM-bypass! [ 3.421934] ima: Allocated hash algorithm: sha1 [ 3.423956] ima: No architecture policies found [ 3.426202] evm: Initialising EVM extended attributes: [ 3.428376] evm: security.selinux [ 3.429777] evm: security.ima [ 3.431273] evm: security.capability [ 3.432576] evm: HMAC attrs: 0x1 [ 3.435277] rtc_cmos 00:05: setting system clock to 2026-08-23 04:56:51 UTC (1787461011) [ 3.442292] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 3.445977] debug: unmapping init [mem 0xffffffff91182000-0xffffffff91458fff] [ 3.456678] Write protecting the kernel read-only data: 28672k [ 3.460537] debug: unmapping init [mem 0xffffffff8f803000-0xffffffff8f9fffff] [ 3.463695] debug: unmapping init [mem 0xffffffff90114000-0xffffffff901fffff] [ 3.496170] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.503423] systemd[1]: Detected virtualization kvm. [ 3.505351] systemd[1]: Detected architecture x86-64. [ 3.507462] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.534484] systemd[1]: No hostname configured. [ 3.536315] systemd[1]: Set hostname to . [ 3.538629] random: systemd: uninitialized urandom read (16 bytes read) [ 3.541238] systemd[1]: Initializing machine ID from random generator. [ 3.581404] random: ln: uninitialized urandom read (6 bytes read) [ 3.708351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.711288] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.716683] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.720652] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.393031] device-mapper: uevent: version 1.0.3 [ 4.394821] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.096311] virtio_net virtio0 ens2: renamed from eth0 [ 5.107903] random: fast init done [ 5.168195] scsi host0: ata_piix [ 5.174268] scsi host1: ata_piix [ 5.250762] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.252928] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.109062] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.942574] random: crng init done [ 9.943842] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.515617] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.767335] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.996357] SELinux: Disabled at runtime. [ 12.051272] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.059508] systemd[1]: Detected virtualization kvm. [ 12.061410] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.526817] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.529951] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.535160] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.540364] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.544745] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.552083] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.558480] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 12.679693] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.062720] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.416092] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.486115] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.611802] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.624897] EDAC sbridge: Ver: 1.1.2 [ 15.194235] Key type dns_resolver registered [ 15.498130] NFS: Registering the id_resolver key type [ 15.500860] Key type id_resolver registered [ 15.502963] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 55.218732] libcfs: loading out-of-tree module taints kernel. [ 55.287873] Key type ._llcrypt registered [ 55.294233] Key type .llcrypt registered [ 55.500056] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 76.476598] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 78.794906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.811396] alg: No test for adler32 (adler32-zlib) [ 80.579322] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 82.146315] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 83.939818] Key type lgssc registered [ 85.638414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.482830] hrtimer: interrupt took 9820677 ns [ 107.790047] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.910761] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 168.329241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.389803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.667559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.687562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.761774] Lustre: lustre-MDT0000: new disk, initializing [ 169.836609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.853891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.994758] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.220325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.348813] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.417736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.436212] Lustre: Skipped 1 previous similar message [ 190.609671] Lustre: lustre-MDT0001: new disk, initializing [ 190.664473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.698043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.716545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.505685] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.910238] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 211.818330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.083638] Lustre: lustre-OST0000: new disk, initializing [ 212.087664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.094430] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.182390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.326332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.338374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.379968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 218.938427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.738516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.939950] Lustre: lustre-OST0001: new disk, initializing [ 236.948027] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.958097] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.079575] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.749034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.760906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.862492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.312846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.538908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.415683] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.630219] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.517993] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 290.276799] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 293.912327] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 297.412811] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 299.831101] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 01:01:47 EDT 2026 [ 321.267252] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 322.972158] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.454470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.473815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.490129] Lustre: Skipped 3 previous similar messages [ 345.568873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.576802] Lustre: Skipped 6 previous similar messages [ 350.691501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.710512] Lustre: Skipped 3 previous similar messages [ 354.271413] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.677777] Lustre: server umount lustre-MDT0000 complete [ 358.983111] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461367 with bad export cookie 10862084223789337463 [ 358.986502] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.999753] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 360.937596] LustreError: 10403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.948277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.953912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.970752] LustreError: 10403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 361.018310] Lustre: Skipped 1 previous similar message [ 363.940385] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.964046] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 365.646776] Lustre: server umount lustre-MDT0001 complete [ 370.164330] Lustre: server umount lustre-OST0000 complete [ 374.908768] Lustre: server umount lustre-OST0001 complete [ 383.274104] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 393.475828] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 436.821112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.164511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.231792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.365062] Lustre: lustre-MDT0000: new disk, initializing [ 437.454197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.503482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.801661] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 456.904084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.074716] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.129560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 457.132885] Lustre: Skipped 1 previous similar message [ 457.188399] Lustre: lustre-MDT0001: new disk, initializing [ 457.280362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 457.315219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 457.327871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 462.014142] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 473.843662] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 484.063428] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.246490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 492.667402] Lustre: lustre-OST0000: new disk, initializing [ 492.671292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 492.676829] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 492.834772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.454119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.472351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 494.565341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.816646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 511.302386] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 517.436847] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.857228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 523.972898] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.308473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 530.416000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.420496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.434808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.448605] Lustre: Skipped 1 previous similar message [ 530.475556] Lustre: Skipped 4 previous similar messages [ 533.231965] Lustre: server umount lustre-OST0000 complete [ 539.115537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.138792] LustreError: Skipped 1 previous similar message [ 539.151393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.178127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.663445] Lustre: server umount lustre-MDT0000 complete [ 545.764266] LustreError: 15098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.791747] LustreError: 15098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 549.718103] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461557 with bad export cookie 10862084223789338996 [ 549.723438] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.749679] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 550.154776] Lustre: server umount lustre-MDT0001 complete [ 555.842552] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:06:03 (1787461563) [ 558.711991] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 560.632214] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:06:08 (1787461568) [ 568.891560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.314815] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.401947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.299193] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 574.433685] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.323405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.682975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 584.701426] Lustre: Skipped 2 previous similar messages [ 590.956126] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 600.932656] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.958956] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.142306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 624.922131] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 634.591913] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 649.721539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 649.722905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.731342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 649.767638] Lustre: Skipped 2 previous similar messages [ 655.639378] Lustre: server umount lustre-OST0000 complete [ 661.990461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.010594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.572469] Lustre: server umount lustre-MDT0000 complete [ 670.177897] LustreError: 20876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.194570] LustreError: 20876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 671.334605] LustreError: 19453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461679 with bad export cookie 10862084223789339878 [ 671.364598] LustreError: 19453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 671.366072] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.886290] Lustre: server umount lustre-MDT0001 complete [ 686.901989] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 690.522956] Key type lgssc unregistered [ 690.793081] LNet: 23033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.813490] LNetError: 23033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.836445] LNet: Removed LNI 192.168.202.102@tcp [ 692.243379] Key type .llcrypt unregistered [ 692.245086] Key type ._llcrypt unregistered [ 725.365894] Key type ._llcrypt registered [ 725.368194] Key type .llcrypt registered [ 725.514836] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 727.267515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.304109] alg: No test for adler32 (adler32-zlib) [ 728.501435] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 728.782742] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 730.511208] Key type lgssc registered [ 732.083907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 742.417809] loop: module loaded [ 742.528189] loop0: detected capacity change from 0 to 204800000 [ 742.842241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.855876] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 743.523770] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.297060] loop0: detected capacity change from 0 to 204800000 [ 751.523569] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.541552] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.558280] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 757.915496] loop0: detected capacity change from 0 to 204800000 [ 764.431499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 764.477874] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.959269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 766.040559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 766.181549] Lustre: lustre-MDT0000: new disk, initializing [ 766.391255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.431403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 771.765964] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 782.360981] loop1: detected capacity change from 0 to 204800000 [ 789.104899] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 789.488607] Lustre: lustre-OST07c6: new disk, initializing [ 789.495617] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 789.506594] Lustre: Skipped 1 previous similar message [ 789.513877] Lustre: 26220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 789.620524] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 796.782139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 796.796364] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 796.836432] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 797.598413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 804.743755] Lustre: 25208:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 804.766971] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 804.783710] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 804.799280] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 804.808267] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 804.817091] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 824.354404] Lustre: 25207:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 824.376852] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 824.395623] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.409456] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 824.425944] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 824.442081] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.080266] Lustre: 25857:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 830.085310] Lustre: 25857:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 830.113706] Lustre: 25857:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 830.126040] Lustre: 25857:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.135991] Lustre: 25857:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.145796] Lustre: 25857:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.152081] Lustre: 25857:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 830.162406] Lustre: 25857:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.169832] Lustre: 25857:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 830.183448] Lustre: 25857:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.189646] Lustre: 25857:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.197029] Lustre: 25857:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.458483] Lustre: 25857:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 833.472040] Lustre: 25857:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 833.484634] Lustre: 25857:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 833.495310] Lustre: 25857:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.505039] Lustre: 25857:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 833.518702] Lustre: 25857:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.531603] Lustre: 25857:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 833.543932] Lustre: 25857:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.554476] Lustre: 25857:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 833.564745] Lustre: 25857:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.578773] Lustre: 25857:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 833.587778] Lustre: 25857:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 842.675622] Lustre: 25208:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 842.709637] Lustre: 25208:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 842.720822] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 842.726733] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.735638] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 842.741914] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.752567] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 842.789395] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.807455] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 842.826543] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.850679] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 842.869996] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.254630] Lustre: 25207:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 851.271182] Lustre: 25207:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 851.283814] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 851.298673] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.310809] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 851.327900] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.342594] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 851.354649] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.363632] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 851.375694] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.391118] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.405636] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 904.162463] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.201793] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.255713] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 909.281132] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 914.400553] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 914.406161] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 914.614997] Lustre: server umount lustre-OST07c6 complete [ 929.266289] Lustre: server umount lustre-MDT0000 complete [ 950.687808] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 954.757272] Key type lgssc unregistered [ 955.245960] LNet: 28783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.256048] LNetError: 28783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.276261] LNet: Removed LNI 192.168.202.102@tcp [ 956.469544] Key type .llcrypt unregistered [ 956.471134] Key type ._llcrypt unregistered [ 980.413182] Key type ._llcrypt registered [ 980.414463] Key type .llcrypt registered [ 980.632052] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 981.476664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.513145] alg: No test for adler32 (adler32-zlib) [ 982.416963] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 982.656216] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 984.346323] Key type lgssc registered [ 985.639905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.869601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.890438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.547113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.021544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1008.100629] LustreError: 30057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.134179] LustreError: 30057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1013.220987] LustreError: 30058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.534773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.024434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.319886] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1032.218627] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1039.061045] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1046.021358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1046.318123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.969147] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1063.594923] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1069.889681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1085.764517] Lustre: server umount lustre-OST0000 complete [ 1090.535403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.536498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.558815] Lustre: Skipped 1 previous similar message [ 1095.652265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.656217] Lustre: Skipped 1 previous similar message [ 1100.777842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.809728] Lustre: Skipped 1 previous similar message [ 1100.966501] Lustre: server umount lustre-MDT0000 complete [ 1105.473409] LustreError: 30036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462113 with bad export cookie 14027800775216029344 [ 1105.479656] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1115.846994] Lustre: server umount lustre-MDT0001 complete [ 1128.139127] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1131.323243] Key type lgssc unregistered [ 1131.589593] LNet: 33800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.599466] LNetError: 33800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.648031] LNet: Removed LNI 192.168.202.102@tcp [ 1133.335555] Key type .llcrypt unregistered [ 1133.337285] Key type ._llcrypt unregistered [ 1155.366232] Key type ._llcrypt registered [ 1155.376828] Key type .llcrypt registered [ 1155.509401] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1157.312923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.377119] alg: No test for adler32 (adler32-zlib) [ 1158.657274] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1158.999958] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1160.695774] Key type lgssc registered [ 1161.962470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.434881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.457428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.309060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.091270] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1184.738614] LustreError: 35073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.759927] LustreError: 35073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1185.769668] LustreError: 35072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.888288] LustreError: 35073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.156553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.814399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.768501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1209.344637] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1217.441660] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1226.259480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.731553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.446841] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1243.822659] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.204107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1263.584728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.593832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.604708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.641265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.645267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.682341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.457112] Lustre: server umount lustre-OST0000 complete [ 1275.874604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.897423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.905387] Lustre: Skipped 2 previous similar messages [ 1278.665980] Lustre: server umount lustre-MDT0000 complete [ 1280.994474] LustreError: 35802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.024733] LustreError: 35802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1283.711736] LustreError: 35053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462291 with bad export cookie 7870714856619293189 [ 1283.722192] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1283.739323] LustreError: 35053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1284.242330] Lustre: server umount lustre-MDT0001 complete [ 1297.675341] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1300.925950] Key type lgssc unregistered [ 1301.390819] LNet: 38635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.403173] LNetError: 38635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.427453] LNet: Removed LNI 192.168.202.102@tcp [ 1302.342789] Key type .llcrypt unregistered [ 1302.344960] Key type ._llcrypt unregistered [ 1331.526950] Key type ._llcrypt registered [ 1331.529356] Key type .llcrypt registered [ 1331.661765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1332.966874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.067425] alg: No test for adler32 (adler32-zlib) [ 1334.364711] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1334.703314] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1336.439255] Key type lgssc registered [ 1337.766947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.051125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.065669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.660354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.262881] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1358.821203] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.867149] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1363.945105] LustreError: 40053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.725488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.182256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.608784] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1383.700079] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1390.984644] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.694336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.335332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.290316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1418.858498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1419.776930] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1428.103309] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1428.450742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.468454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.964321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.979562] Lustre: Skipped 2 previous similar messages [ 1433.178087] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1433.558611] Lustre: server umount lustre-MDT0000 complete [ 1434.094038] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.120374] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1438.130253] LustreError: 40032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462446 with bad export cookie 980208356134459969 [ 1438.142229] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.163475] LustreError: 40032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.295747] LustreError: 41038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.318397] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 1439.204289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.209979] Lustre: Skipped 2 previous similar messages [ 1443.233736] LustreError: 40046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.233885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1443.254647] LustreError: 40046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1443.273648] Lustre: Skipped 1 previous similar message [ 1444.794971] Lustre: server umount lustre-MDT0001 complete [ 1475.039797] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1475.413240] Lustre: server umount lustre-OST0000 complete [ 1497.272899] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1501.947392] Key type lgssc unregistered [ 1502.422086] LNet: 43924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.455680] LNetError: 43924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.480518] LNet: Removed LNI 192.168.202.102@tcp [ 1503.549397] Key type .llcrypt unregistered [ 1503.551587] Key type ._llcrypt unregistered [ 1534.633881] Key type ._llcrypt registered [ 1534.636774] Key type .llcrypt registered [ 1534.814419] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1535.718540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.829514] alg: No test for adler32 (adler32-zlib) [ 1537.364524] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1537.779811] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1539.578342] Key type lgssc registered [ 1541.079143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1550.364248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1550.389761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.848550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.019783] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1561.060728] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.082352] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1566.188476] LustreError: 45344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.450236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.924155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1573.588914] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1582.135571] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1590.873105] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1600.503592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.245953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.713941] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1621.070038] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.827860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1632.225160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.230383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.237217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.257551] Lustre: Skipped 1 previous similar message [ 1636.999289] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1637.015819] Lustre: Skipped 1 previous similar message [ 1638.427546] Lustre: server umount lustre-OST0000 complete [ 1660.900451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1660.918588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.970573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.980593] Lustre: Skipped 2 previous similar messages [ 1661.991034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.279143] Lustre: server umount lustre-MDT0000 complete [ 1667.054502] LustreError: 45344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.086045] LustreError: 45344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1670.185278] LustreError: 45326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462678 with bad export cookie 3639160832574213481 [ 1670.190910] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.200303] LustreError: 45326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1676.809957] Lustre: server umount lustre-MDT0001 complete [ 1689.720456] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1693.456439] Key type lgssc unregistered [ 1693.918739] LNet: 49059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.927102] LNetError: 49059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.953363] LNet: Removed LNI 192.168.202.102@tcp [ 1695.225726] Key type .llcrypt unregistered [ 1695.228261] Key type ._llcrypt unregistered [ 1725.491538] Key type ._llcrypt registered [ 1725.497515] Key type .llcrypt registered [ 1725.717692] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1727.397618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.595589] alg: No test for adler32 (adler32-zlib) [ 1728.865693] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1729.235103] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1730.923848] Key type lgssc registered [ 1732.336819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.555979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1741.578870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.159911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.108835] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1753.587949] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.597150] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1758.695646] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.697850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.052760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1765.858294] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1776.163311] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1784.411626] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.500580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.805262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.702860] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1811.014788] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1813.321099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1821.290601] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1825.446940] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1838.045655] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-23 1h27m24s [ 1839.986990] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1842.144478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1842.154762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.168659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.236422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.249338] Lustre: Skipped 5 previous similar messages [ 1848.406732] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.966519] Lustre: server umount lustre-MDT0000 complete [ 1853.444112] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1854.547423] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.580678] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1860.705521] LustreError: 51426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.723455] LustreError: 51426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1862.250358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.398423] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.678431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.758372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1863.744242] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1868.309850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1872.894878] Lustre: DEBUG MARKER: Wait for df (36780) ... [ 1875.565838] Lustre: DEBUG MARKER: done [ 1883.106947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.120187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.135183] Lustre: Skipped 2 previous similar messages [ 1883.141395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.144408] Lustre: Skipped 1 previous similar message [ 1884.142600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.566898] Lustre: server umount lustre-OST0000 complete [ 1894.369724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1894.370914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.386326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.386335] Lustre: Skipped 2 previous similar messages [ 1894.412297] Lustre: Skipped 1 previous similar message [ 1899.247699] Lustre: server umount lustre-MDT0000 complete [ 1901.537114] LustreError: 51426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.566915] LustreError: 51426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1903.969652] LustreError: 50459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462912 with bad export cookie 5062555024368429417 [ 1903.989422] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.396939] Lustre: server umount lustre-MDT0001 complete [ 1918.125266] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1922.040089] Key type lgssc unregistered [ 1922.614531] LNet: 55426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.621361] LNetError: 55426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.641255] LNet: Removed LNI 192.168.202.102@tcp [ 1923.961382] Key type .llcrypt unregistered [ 1923.969732] Key type ._llcrypt unregistered [ 1975.349413] Key type ._llcrypt registered [ 1975.351497] Key type .llcrypt registered [ 1975.430453] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1977.452806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.476458] alg: No test for adler32 (adler32-zlib) [ 1978.550610] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1978.908689] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1980.650220] Key type lgssc registered [ 1981.989341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.007783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.032047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.353229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1992.389804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1992.639926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.871774] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2009.137338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.251054] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2009.292755] Lustre: 57635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2009.675749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.980913] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2024.021480] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.387876] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.143044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2038.436896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2038.801365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.522693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2046.678647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2053.098370] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2061.031259] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-23 1h31m08s [ 2062.838580] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2073.126377] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2076.581140] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2076.635512] LustreError: 57641:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2096.798591] Lustre: lustre-MDT0000: Client e038659b-2e70-42c0-8008-ed6af4fa47c8 (at 192.168.202.2@tcp) reconnecting [ 2116.679159] LustreError: 57641:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2119.511400] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2122.864279] Lustre: DEBUG MARKER: done [ 2133.986511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2134.002075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.012237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.022622] Lustre: Skipped 1 previous similar message [ 2136.469270] Lustre: server umount lustre-OST0000 complete [ 2142.688863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2142.698541] LustreError: Skipped 1 previous similar message [ 2142.703920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.715966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.722156] Lustre: Skipped 1 previous similar message [ 2143.776785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.784533] Lustre: Skipped 2 previous similar messages [ 2145.828573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.842368] Lustre: Skipped 1 previous similar message [ 2147.615199] Lustre: server umount lustre-MDT0000 complete [ 2151.940619] LustreError: 59497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463160 with bad export cookie 14710167329631409897 [ 2151.946097] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.953128] LustreError: 59497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2152.513316] Lustre: server umount lustre-MDT0001 complete [ 2167.256416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2171.590947] Key type lgssc unregistered [ 2171.973737] LNet: 61968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.980064] LNetError: 61968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.059160] LNet: Removed LNI 192.168.202.102@tcp [ 2174.409851] Key type .llcrypt unregistered [ 2174.413355] Key type ._llcrypt unregistered [ 2237.419273] Key type ._llcrypt registered [ 2237.423196] Key type .llcrypt registered [ 2237.593285] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2238.407923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.465982] alg: No test for adler32 (adler32-zlib) [ 2239.744340] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2240.258053] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2241.951162] Key type lgssc registered [ 2244.860770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.529951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.603080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.032242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2254.072604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2254.270133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.872639] Lustre: Failing over lustre-MDT0000 [ 2257.160948] Lustre: server umount lustre-MDT0000 complete [ 2266.374370] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:34:33 (1787463273) [ 2275.536101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.109553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.835305] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2292.575551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.714631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2292.782855] Lustre: 64770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2293.069256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2298.296361] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2308.616507] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2315.938823] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.097285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.240227] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2325.466868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.242695] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2334.900881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2341.855955] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2347.714374] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2359.805495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.811855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.833507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.803830] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.805928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.841236] Lustre: Skipped 1 previous similar message [ 2364.529575] Lustre: server umount lustre-MDT0000 complete [ 2369.336676] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463377 with bad export cookie 334482500938439190 [ 2369.345407] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.738846] Lustre: server umount lustre-MDT0001 complete [ 2374.908958] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2383.298440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2387.232403] Lustre: 63693:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463379/real 1787463379] req@00000000fdd47056 x1874291062778368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2387.302469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.344871] Lustre: Skipped 1 previous similar message [ 2390.559344] Lustre: 63692:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463382/real 1787463382] req@000000007277cd6d x1874291062778496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2392.545396] Lustre: 63691:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463384/real 1787463384] req@00000000394a22b7 x1874291062778624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2395.615200] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463387/real 1787463387] req@00000000265ee229 x1874291062778880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2396.218442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.799275] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463392/real 1787463392] req@000000006caa51ba x1874291062779008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2414.339375] LustreError: 66620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.365049] LustreError: 66620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2414.418712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.522633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2416.416948] LustreError: 66621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.451132] LustreError: 66620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.454664] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2420.026676] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2422.758561] LustreError: 66621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.875808] LustreError: 66620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.352185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.885531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.872098] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2437.265508] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2445.832156] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2452.523989] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2462.690071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2462.704745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2462.727966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.428493] Lustre: server umount lustre-MDT0000 complete [ 2468.838680] LustreError: 68544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.858725] LustreError: 68544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2472.040395] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463480 with bad export cookie 334482500938441465 [ 2472.051095] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2472.063892] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.461855] Lustre: server umount lustre-MDT0001 complete [ 2482.044270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2490.335879] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463482/real 1787463482] req@00000000dac889c7 x1874291062809088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2490.375718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.396126] Lustre: Skipped 2 previous similar messages [ 2493.243283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.569981] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4a4520ffc05285c [ 2497.591810] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 2498.036350] LustreError: 66620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.135919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.201778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2504.168996] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2513.733780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.479374] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2519.932687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2528.950381] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.896363] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2543.074753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.084476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.096215] Lustre: Skipped 2 previous similar messages [ 2543.110848] Lustre: Skipped 5 previous similar messages [ 2548.200347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.760420] Lustre: server umount lustre-MDT0000 complete [ 2553.320297] LustreError: 72186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.354183] LustreError: 72186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2553.579554] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463561 with bad export cookie 334482500938442844 [ 2553.580936] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.604077] LustreError: 64761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2553.937191] Lustre: server umount lustre-MDT0001 complete [ 2562.655923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2566.850745] Lustre: DEBUG MARKER: files compared the same [ 2578.417561] Lustre: server umount lustre-OST0000 complete [ 2598.518765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2601.299655] Key type lgssc unregistered [ 2601.597148] LNet: 74508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.605725] LNetError: 74508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2601.622574] LNet: Removed LNI 192.168.202.102@tcp [ 2602.889602] Key type .llcrypt unregistered [ 2602.892598] Key type ._llcrypt unregistered [ 2630.484791] Key type ._llcrypt registered [ 2630.486662] Key type .llcrypt registered [ 2630.712480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2632.506625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.644981] alg: No test for adler32 (adler32-zlib) [ 2634.068424] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2634.556855] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2636.375305] Key type lgssc registered [ 2638.185871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.151059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.171156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.760424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.004518] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2659.297286] LustreError: 75930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.327487] LustreError: 75930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2664.416850] LustreError: 75931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.460593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.728958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2671.466000] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2681.545292] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2689.015802] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2697.634058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.111067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.295266] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2715.217742] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2719.601834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2729.952370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.955238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.964746] Lustre: Skipped 1 previous similar message [ 2729.977308] Lustre: Skipped 1 previous similar message [ 2735.073806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.078690] Lustre: Skipped 1 previous similar message [ 2736.347595] Lustre: server umount lustre-OST0000 complete [ 2743.775903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2743.780968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.795789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.314366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.015629] Lustre: server umount lustre-MDT0000 complete [ 2750.432038] LustreError: 77949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.455544] LustreError: 77949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2753.309608] LustreError: 75912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463761 with bad export cookie 993614256342487993 [ 2753.334603] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.336646] LustreError: 75912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.654160] Lustre: server umount lustre-MDT0001 complete [ 2767.042846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2769.918975] Key type lgssc unregistered [ 2770.271508] LNet: 79490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2770.284901] LNetError: 79490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2770.301450] LNet: Removed LNI 192.168.202.102@tcp [ 2771.307871] Key type .llcrypt unregistered [ 2771.311233] Key type ._llcrypt unregistered [ 2804.549103] Key type ._llcrypt registered [ 2804.559324] Key type .llcrypt registered [ 2804.856588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2806.497876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2806.621686] alg: No test for adler32 (adler32-zlib) [ 2808.117753] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2808.628578] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2810.359430] Key type lgssc registered [ 2812.261515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.583562] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.612842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2839.135553] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2844.255967] LustreError: 80888:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 2870.946336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.100798] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2886.924995] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2895.353741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.571542] Lustre: *** cfs_fail_loc=706, val=0*** [ 2896.093218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.166058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2901.063568] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2901.476158] LustreError: 80914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.489585] LustreError: 80914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2906.624472] LustreError: 80913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.641467] LustreError: 80913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2910.723464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.286833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2917.144230] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2925.315896] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2932.078908] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.824112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.829708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2936.849279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2936.869247] Lustre: Skipped 1 previous similar message [ 2941.457502] Lustre: server umount lustre-OST0000 complete [ 2947.041491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.044567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.054407] Lustre: Skipped 2 previous similar messages [ 2947.066755] Lustre: Skipped 1 previous similar message [ 2952.160071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.167084] Lustre: Skipped 1 previous similar message [ 2953.033506] Lustre: server umount lustre-MDT0000 complete [ 2956.266799] LustreError: 81972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.312738] LustreError: 81972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2957.351600] LustreError: 80896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463965 with bad export cookie 17917003799075013017 [ 2957.367731] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.136556] Lustre: server umount lustre-MDT0001 complete [ 2971.602703] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2974.479574] Key type lgssc unregistered [ 2974.740815] LNet: 84508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.746751] LNetError: 84508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2974.757237] LNet: Removed LNI 192.168.202.102@tcp [ 2975.527821] Key type .llcrypt unregistered [ 2975.531611] Key type ._llcrypt unregistered [ 3011.282250] Key type ._llcrypt registered [ 3011.287321] Key type .llcrypt registered [ 3011.485037] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3012.567890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3012.591995] alg: No test for adler32 (adler32-zlib) [ 3013.962472] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 3014.343971] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3016.055217] Key type lgssc registered [ 3017.519209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3027.570259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3027.598787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.680788] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3047.662932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.670766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3054.064805] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.088045] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3056.159586] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3058.159877] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.197717] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3063.273154] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.301960] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3067.764659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.080095] LustreError: 86623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.112652] LustreError: 86623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3068.130718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.250630] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.270988] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3076.009178] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3083.755446] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.800381] LustreError: 86620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3086.345400] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.350323] Lustre: MGS: Client a2a2ebb5-3e84-431f-a92e-c2671f31856f (at 0@lo) reconnecting [ 3086.365101] LustreError: 86634:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3086.518878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.588678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3092.181614] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3107.302396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.305377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.323587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3107.336221] Lustre: Skipped 1 previous similar message [ 3112.442073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3112.452055] Lustre: Skipped 2 previous similar messages [ 3113.342609] Lustre: server umount lustre-OST0000 complete [ 3117.536989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.540619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3117.569310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.579336] Lustre: Skipped 1 previous similar message [ 3122.656817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.669569] Lustre: Skipped 1 previous similar message [ 3123.770945] Lustre: server umount lustre-MDT0000 complete [ 3127.797841] LustreError: 88405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.819410] LustreError: 88405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3128.203581] Lustre: server umount lustre-MDT0001 complete [ 3131.916505] Lustre: server umount MGS complete [ 3147.883363] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:49:14 (1787464154) [ 3185.242294] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3196.352860] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3238.354088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.273481] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3255.991423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.307110] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3256.357728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3256.474681] Lustre: lustre-MDT0001: new disk, initializing [ 3256.563052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3256.634844] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3256.656658] Lustre: 93816:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3256.674856] LustreError: 93816:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3261.358480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3273.015908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.328771] Lustre: lustre-OST0000: new disk, initializing [ 3273.346521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3273.371340] Lustre: 94583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3273.439360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.254927] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3280.481084] Lustre: 94877:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000064f0c9ba x1874291874375296/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787464304 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3293.132924] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3300.831952] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3302.056384] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.083566] Lustre: MGS: Client 87ee7d59-1d28-4b2c-8ca8-794c63ac67c0 (at 0@lo) reconnecting [ 3302.142515] Lustre: 93086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3302.180212] LustreError: 93802:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3302.196630] LustreError: 93802:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3302.224316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3302.354375] Lustre: lustre-MDT0000: new disk, initializing [ 3302.637399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.669343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3303.722356] Lustre: 95653:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001e78a047 x1874291874382720/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787464327 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3307.551917] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3308.770418] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3312.822812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3312.833155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3312.924749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3341.802785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.843817] Lustre: Skipped 2 previous similar messages [ 3341.853990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.469924] Lustre: server umount lustre-OST0000 complete [ 3352.033390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.072920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.077594] Lustre: Skipped 2 previous similar messages [ 3364.319182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3364.459214] Lustre: server umount lustre-MDT0000 complete [ 3366.380172] LustreError: 93788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.409508] LustreError: 93788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3375.797631] Lustre: server umount lustre-MDT0001 complete [ 3380.192245] Lustre: server umount MGS complete [ 3394.949978] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:53:22 (1787464402) [ 3416.048558] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3418.939073] Key type lgssc unregistered [ 3419.396494] LNet: 98157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3419.401612] LNetError: 98157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3419.419316] LNet: Removed LNI 192.168.202.102@tcp [ 3420.943839] Key type .llcrypt unregistered [ 3420.947652] Key type ._llcrypt unregistered [ 3444.250810] Key type ._llcrypt registered [ 3444.254283] Key type .llcrypt registered [ 3444.432716] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3446.315635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3446.541185] alg: No test for adler32 (adler32-zlib) [ 3447.962596] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 3448.309778] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3450.135454] Key type lgssc registered [ 3451.673299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3462.954846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3462.972304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.790130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.767353] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3474.913913] LustreError: 99381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.933319] LustreError: 99381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3480.033781] LustreError: 99382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.396553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.628841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.808765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3498.526523] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.207246] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3516.159589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.691913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.979916] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3533.135841] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3534.061451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3548.642441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.644747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.664849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.665086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.666889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3549.683844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.218289] Lustre: server umount lustre-OST0000 complete [ 3559.396861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3559.409444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.427076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.439364] Lustre: Skipped 1 previous similar message [ 3565.039747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.047493] Lustre: Skipped 2 previous similar messages [ 3565.429794] Lustre: server umount lustre-MDT0000 complete [ 3569.786191] LustreError: 100118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464577 with bad export cookie 17104543042728877285 [ 3569.787456] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.793039] LustreError: 100118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3570.214378] Lustre: server umount lustre-MDT0001 complete [ 3583.361978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3586.933967] Key type lgssc unregistered [ 3587.280481] LNet: 102942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3587.287761] LNetError: 102942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.302601] LNet: Removed LNI 192.168.202.102@tcp [ 3588.437177] Key type .llcrypt unregistered [ 3588.442227] Key type ._llcrypt unregistered [ 3609.663310] Key type ._llcrypt registered [ 3609.665281] Key type .llcrypt registered [ 3609.867923] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3612.295388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.339428] alg: No test for adler32 (adler32-zlib) [ 3613.563705] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 3613.804793] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3615.503294] Key type lgssc registered [ 3617.522763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3626.152124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3626.152913] LustreError: 104034:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3626.154812] Lustre: Skipped 1 previous similar message [ 3626.154829] LustreError: 104035:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3626.168175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.173671] LustreError: 104034:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3626.176575] LustreError: 104034:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3626.189507] LustreError: 104034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3627.506051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.729856] LustreError: 104073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.752213] LustreError: 104073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3638.001876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.308874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3643.966569] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3651.143582] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lsmod [ 3655.214782] LustreError: 105240:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3655.215962] LustreError: 105243:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3655.228792] LustreError: 105240:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3655.240387] LustreError: 105240:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3655.245439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3655.251592] LustreError: 105240:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3655.700369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3660.770321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.791457] Lustre: Skipped 1 previous similar message [ 3660.794233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.832861] Lustre: Skipped 2 previous similar messages [ 3663.913402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.938449] Lustre: Skipped 2 previous similar messages [ 3666.691724] Lustre: server umount lustre-MDT0000 complete [ 3668.963940] LustreError: 104717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.992957] LustreError: 104717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3671.863763] LustreError: 104048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464679 with bad export cookie 18126033702601442338 [ 3671.891644] LustreError: 104048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3671.893144] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.184327] LustreError: 104505:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004b7161e0 x1874292502451200/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3672.422509] Lustre: server umount lustre-MDT0001 complete [ 3688.019031] Lustre: server umount lustre-OST0000 complete [ 3700.337759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.215975] LustreError: 106308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.318824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.344909] LustreError: 106309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.897996] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3711.468317] LustreError: 106308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.244420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.679610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3726.632631] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3737.646411] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.832530] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3755.246166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.882495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.351160] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3771.285109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3777.067556] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3786.721852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.740509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.745084] Lustre: Skipped 1 previous similar message [ 3786.752673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.846085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.857265] Lustre: Skipped 2 previous similar messages [ 3792.392512] Lustre: server umount lustre-OST0000 complete [ 3802.097945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.101918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.122299] Lustre: Skipped 2 previous similar messages [ 3804.283913] Lustre: server umount lustre-MDT0000 complete [ 3807.207365] LustreError: 106304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.227571] LustreError: 106304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3809.923212] LustreError: 109094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464817 with bad export cookie 18126033702601443038 [ 3809.925473] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.935404] LustreError: 109094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3810.511983] Lustre: server umount lustre-MDT0001 complete [ 3830.154789] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3832.841882] Key type lgssc unregistered [ 3833.211861] LNet: 109872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.235653] LNetError: 109872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.269200] LNet: Removed LNI 192.168.202.102@tcp [ 3835.045215] Key type .llcrypt unregistered [ 3835.049980] Key type ._llcrypt unregistered [ 3869.225218] Key type ._llcrypt registered [ 3869.227863] Key type .llcrypt registered [ 3869.364485] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3870.861060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.081904] alg: No test for adler32 (adler32-zlib) [ 3872.434799] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 3872.746252] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3874.433471] Key type lgssc registered [ 3876.185974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3887.546354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3887.566111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.066635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.840051] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3899.366404] LustreError: 111299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.410958] LustreError: 111299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3904.488523] LustreError: 111300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.077445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.621824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.732989] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3922.525491] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.418624] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.359673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.934066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.877122] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3959.349320] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3959.486243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3966.310824] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3971.642694] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3977.193293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3977.206805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.226265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.257493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.260354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.271075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.299479] Lustre: Skipped 1 previous similar message [ 3983.169828] Lustre: server umount lustre-OST0000 complete [ 3988.448335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.458917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.473177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3990.516044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3990.517221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.523586] Lustre: server umount lustre-MDT0000 complete [ 3995.619115] LustreError: 111296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.654318] LustreError: 111296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3999.917100] LustreError: 111280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465007 with bad export cookie 5638026653059219021 [ 3999.937609] LustreError: 111280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3999.938661] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.567671] Lustre: server umount lustre-MDT0001 complete [ 4016.086707] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4019.489239] Key type lgssc unregistered [ 4019.803588] LNet: 114948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4019.812521] LNetError: 114948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4019.852291] LNet: Removed LNI 192.168.202.102@tcp [ 4020.978871] Key type .llcrypt unregistered [ 4020.982250] Key type ._llcrypt unregistered [ 4043.506510] Key type ._llcrypt registered [ 4043.511752] Key type .llcrypt registered [ 4043.707640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4045.128248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.147509] alg: No test for adler32 (adler32-zlib) [ 4046.546647] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 4046.824268] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4048.495212] Key type lgssc registered [ 4050.049737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4060.654331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4060.670829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.266591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.181353] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4072.419184] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.450232] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4077.542053] LustreError: 116170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.109496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.590574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4085.459969] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4095.168304] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4102.649254] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4111.214923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4111.621407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.817847] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4128.371793] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4132.224741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4147.683177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.688581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.688743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4147.717884] Lustre: Skipped 1 previous similar message [ 4148.757793] Lustre: server umount lustre-OST0000 complete [ 4156.387550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4156.398523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.426319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.429025] Lustre: Skipped 1 previous similar message [ 4157.928290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.930545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.672242] Lustre: server umount lustre-MDT0000 complete [ 4163.045941] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.082792] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4164.749300] LustreError: 116151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465172 with bad export cookie 11028632312210042743 [ 4164.753376] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.756526] LustreError: 116151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4164.951495] Lustre: server umount lustre-MDT0001 complete [ 4182.951188] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4189.081993] Key type lgssc unregistered [ 4189.757202] LNet: 119733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4189.778238] LNetError: 119733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4189.824761] LNet: Removed LNI 192.168.202.102@tcp [ 4191.804863] Key type .llcrypt unregistered [ 4191.817783] Key type ._llcrypt unregistered [ 4228.202464] Key type ._llcrypt registered [ 4228.204406] Key type .llcrypt registered [ 4228.435682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4229.789038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4229.807721] alg: No test for adler32 (adler32-zlib) [ 4231.473909] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 4231.923111] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4233.760247] Key type lgssc registered [ 4235.266926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4247.139690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4247.166710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.949726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.741351] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4258.296311] LustreError: 121170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.324690] LustreError: 121170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4263.392588] LustreError: 121171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.516667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.265808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.554337] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4281.369872] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4289.420565] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4298.198490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.875136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.328714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4318.369368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4318.749483] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4328.583812] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4333.419063] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4333.425379] Lustre: Skipped 2 previous similar messages [ 4335.156696] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4335.203890] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4339.716249] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4339.723418] Lustre: Skipped 2 previous similar messages [ 4340.549907] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4340.557859] Lustre: Skipped 1 previous similar message [ 4349.533906] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4349.547539] Lustre: Skipped 2 previous similar messages [ 4355.681360] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp [ 4361.059963] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.2/24@tcp in log lustre-MDT0000 [ 4361.067375] Lustre: Skipped 2 previous similar messages [ 4361.334266] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 4361.341219] Lustre: Skipped 1 previous similar message [ 4369.900016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4369.903905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.934576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4369.957520] Lustre: Skipped 1 previous similar message [ 4369.979737] Lustre: Skipped 1 previous similar message [ 4374.797242] Lustre: server umount lustre-OST0000 complete [ 4384.225280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4384.243272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.273296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.259956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.277050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.283511] Lustre: Skipped 1 previous similar message [ 4387.691733] Lustre: server umount lustre-MDT0000 complete [ 4390.373966] LustreError: 121170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.406506] LustreError: 121170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4392.813881] LustreError: 124415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465400 with bad export cookie 3615463568969755354 [ 4392.820023] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.826575] LustreError: 124415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4393.284356] Lustre: server umount lustre-MDT0001 complete [ 4407.024957] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4410.578486] Key type lgssc unregistered [ 4410.962896] LNet: 125879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4410.987286] LNetError: 125879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4411.010167] LNet: Removed LNI 192.168.202.102@tcp [ 4411.941898] Key type .llcrypt unregistered [ 4411.946169] Key type ._llcrypt unregistered [ 4450.204646] Key type ._llcrypt registered [ 4450.207954] Key type .llcrypt registered [ 4450.341536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4451.872529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4451.960598] alg: No test for adler32 (adler32-zlib) [ 4452.967426] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 4453.286096] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4455.096158] Key type lgssc registered [ 4456.622169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4465.190284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4465.222488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.964658] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4467.008728] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.2/24@tcp [ 4467.057490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.724964] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4477.411351] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.447857] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4482.531520] LustreError: 127442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.921205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.517204] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4483.562221] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 4483.598630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.846631] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4499.157197] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4506.139366] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4513.537475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.063412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.627272] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4533.466421] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.622422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4553.184721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.184755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4553.199721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.202355] Lustre: Skipped 1 previous similar message [ 4558.309952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4558.318745] Lustre: Skipped 2 previous similar messages [ 4558.742878] Lustre: server umount lustre-OST0000 complete [ 4563.427863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.453422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.463505] Lustre: Skipped 1 previous similar message [ 4568.548573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.473464] Lustre: server umount lustre-MDT0000 complete [ 4573.687503] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.709200] LustreError: 127443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4574.242792] LustreError: 127422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465582 with bad export cookie 230943719495818169 [ 4574.267508] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4574.866391] Lustre: server umount lustre-MDT0001 complete [ 4590.276785] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4593.753893] Key type lgssc unregistered [ 4594.209850] LNet: 131057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4594.216903] LNetError: 131057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4594.238740] LNet: Removed LNI 192.168.202.102@tcp [ 4595.235618] Key type .llcrypt unregistered [ 4595.237446] Key type ._llcrypt unregistered [ 4630.841920] Key type ._llcrypt registered [ 4630.843792] Key type .llcrypt registered [ 4631.017522] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4653.991746] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4655.914315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.965875] alg: No test for adler32 (adler32-zlib) [ 4657.128546] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 4657.371624] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4659.055436] Key type lgssc registered [ 4660.133392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4706.966676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4706.996414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.242245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4708.261238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4708.368965] Lustre: lustre-MDT0000: new disk, initializing [ 4708.532541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.589536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4714.418152] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4727.648602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.845391] Lustre: 134463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4727.944408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4727.951529] Lustre: Skipped 1 previous similar message [ 4728.053806] Lustre: lustre-MDT0001: new disk, initializing [ 4728.153240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4728.239256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4728.256542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4734.109919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4745.887654] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4752.954341] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4761.752925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4762.240591] Lustre: lustre-OST0000: new disk, initializing [ 4762.247289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4762.264580] Lustre: 136431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4762.474908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.920244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4769.946199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4770.140267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4771.176403] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4783.105803] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4789.165766] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4789.477860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4796.007828] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4796.320127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4801.002955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4801.004523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.011694] LustreError: Skipped 1 previous similar message [ 4801.012426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4801.065123] Lustre: Skipped 1 previous similar message [ 4805.437521] Lustre: server umount lustre-OST0000 complete [ 4811.234540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.260548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.266360] Lustre: Skipped 1 previous similar message [ 4816.355086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4816.364236] Lustre: Skipped 2 previous similar messages [ 4816.820669] Lustre: server umount lustre-MDT0000 complete [ 4821.507264] LustreError: 134455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465829 with bad export cookie 5975941650444804191 [ 4821.512349] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.521193] LustreError: 134455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4821.799336] Lustre: server umount lustre-MDT0001 complete [ 4832.770991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.559449] LustreError: 138648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.586884] LustreError: 138648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4833.687116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4838.881448] LustreError: 138649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.683355] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4842.984696] LustreError: 138648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.102576] LustreError: 138649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.764320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.313638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.465489] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4868.779775] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4877.123826] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4885.818983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.403062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.121393] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4906.037932] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.196709] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4912.660706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4919.871356] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4920.197699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4931.361213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.689808] Lustre: lustre-OST0001: new disk, initializing [ 4931.693680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4931.703568] Lustre: 142029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4931.966815] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4940.524504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4940.563374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4940.770545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4941.541463] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4953.158601] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4953.457809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4959.397372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4959.854672] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4969.579629] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa08d08010800.ost_server_uuid 50 [ 4972.537546] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa08d08010800.ost_server_uuid in IDLE state after 0 sec [ 4986.848630] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4986.849874] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.863107] LustreError: Skipped 1 previous similar message [ 4986.884688] Lustre: Skipped 1 previous similar message [ 4986.886542] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4988.931281] Lustre: server umount lustre-OST0001 complete [ 4991.972307] LustreError: 140475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.989683] LustreError: 140475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4995.041185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4995.049898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.070775] Lustre: Skipped 1 previous similar message [ 4995.077098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4995.093651] Lustre: Skipped 1 previous similar message [ 5002.209584] LustreError: 140475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.240802] LustreError: 140475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5007.330472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5007.344055] Lustre: Skipped 6 previous similar messages [ 5008.351212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5008.607128] Lustre: server umount lustre-OST0000 complete [ 5016.032930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5016.054191] LustreError: Skipped 1 previous similar message [ 5016.062252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.081392] Lustre: Skipped 1 previous similar message [ 5020.118792] Lustre: server umount lustre-MDT0000 complete [ 5022.688213] LustreError: 138648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.720943] LustreError: 138648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5024.760894] LustreError: 142028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466032 with bad export cookie 5975941650444805073 [ 5024.764849] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5024.769444] LustreError: 142028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5025.223218] Lustre: server umount lustre-MDT0001 complete [ 5049.601755] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5052.629679] Key type lgssc unregistered [ 5052.964189] LNet: 145051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5052.975629] LNetError: 145051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5052.997223] LNet: Removed LNI 192.168.202.102@tcp [ 5054.318137] Key type .llcrypt unregistered [ 5054.325756] Key type ._llcrypt unregistered [ 5111.716341] Key type ._llcrypt registered [ 5111.719766] Key type .llcrypt registered [ 5111.831714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5134.841843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5136.526241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5136.595072] alg: No test for adler32 (adler32-zlib) [ 5138.000205] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 5138.278887] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5139.929498] Key type lgssc registered [ 5141.816993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.754607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5192.827390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.292300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5194.331838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5194.445799] Lustre: lustre-MDT0000: new disk, initializing [ 5194.655867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.690462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.270680] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5216.783270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.908395] Lustre: 149218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5216.967338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5216.971555] Lustre: Skipped 1 previous similar message [ 5217.050079] Lustre: lustre-MDT0001: new disk, initializing [ 5217.183580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5217.221126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5217.237336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5225.046574] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5236.283240] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.516250] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5252.112726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.485415] Lustre: lustre-OST0000: new disk, initializing [ 5252.493592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5252.501476] Lustre: 151188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5252.638048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.407246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5260.425896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5260.574076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5261.422569] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5274.667859] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5282.262755] Lustre: Failing over lustre-OST0000 [ 5282.491527] Lustre: server umount lustre-OST0000 complete [ 5298.143190] Lustre: 147437:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466290/real 1787466290] req@000000004731a939 x1874294101545600/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787466306 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5298.186890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.241961] Lustre: 147437:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466294/real 1787466294] req@0000000083b7f6fb x1874294101546112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787466310 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5302.277591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.466156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5303.922403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.948951] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5307.883059] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466299/real 1787466299] req@00000000100f9e01 x1874294101547776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787466315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5307.945469] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5308.006879] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5312.032748] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5312.095951] Lustre: 147438:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466304/real 1787466304] req@00000000d588e619 x1874294101549056/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787466320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5312.154964] Lustre: 147438:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5312.218489] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 5312.219521] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5318.152534] Lustre: Failing over lustre-MDT0000 [ 5318.626275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.633136] Lustre: Skipped 2 previous similar messages [ 5318.803281] Lustre: server umount lustre-MDT0000 complete [ 5323.745920] LustreError: 149230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.780625] LustreError: 149230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5324.886072] LustreError: 149224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.868832] LustreError: 151499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.912514] LustreError: 151499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5333.985933] LustreError: 150348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.012644] LustreError: 150348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5339.106391] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466331/real 1787466331] req@0000000088d56182 x1874294101563264/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787466347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5339.120644] LustreError: 149226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.165429] Lustre: 147436:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5339.165547] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.251927] LustreError: 149226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5340.482293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.426309] LustreError: 149225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.465443] LustreError: 149225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5350.371251] LustreError: 147434:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cc6fbc35 x1874294101568384/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5350.682345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.800152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5352.534717] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5355.081166] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5356.022799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5356.035439] Lustre: Skipped 1 previous similar message [ 5356.092790] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5356.180205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5366.243789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.270543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5366.292643] Lustre: Skipped 1 previous similar message [ 5371.394510] LustreError: 152427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.429494] LustreError: 152427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5371.826743] Lustre: server umount lustre-OST0000 complete [ 5381.603369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.639454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5381.642495] Lustre: Skipped 1 previous similar message [ 5386.730420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5386.755943] Lustre: Skipped 1 previous similar message [ 5391.840811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.853686] Lustre: Skipped 1 previous similar message [ 5392.351995] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5392.812417] Lustre: server umount lustre-MDT0000 complete [ 5398.490422] LustreError: 149211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466406 with bad export cookie 6058139733137472303 [ 5398.492058] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5398.508064] LustreError: 149211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5399.450510] Lustre: server umount lustre-MDT0001 complete [ 5414.465341] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5418.579338] Key type lgssc unregistered [ 5418.881210] LNet: 154862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5418.891137] LNetError: 154862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5418.930585] LNet: Removed LNI 192.168.202.102@tcp [ 5420.238920] Key type .llcrypt unregistered [ 5420.241427] Key type ._llcrypt unregistered [ 5469.164306] Key type ._llcrypt registered [ 5469.169682] Key type .llcrypt registered [ 5469.326097] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5471.305073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5471.564231] alg: No test for adler32 (adler32-zlib) [ 5473.071666] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 5473.434945] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5475.143196] Key type lgssc registered [ 5477.006586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5487.702429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5487.739149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.330791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.518535] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5499.872731] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.898731] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5504.993826] LustreError: 156386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.323976] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5510.115283] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.625820] LustreError: 156414:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5512.649201] LustreError: 156414:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5512.679107] LustreError: 156414:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 5513.176857] Lustre: server umount lustre-MDT0000 complete [ 5533.250195] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5549.718140] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5602.916934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.348112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5603.392067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5603.549337] Lustre: lustre-MDT0000: new disk, initializing [ 5603.736322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5603.797582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5610.931896] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5626.367238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.668656] Lustre: 159741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5626.804666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5626.816923] Lustre: Skipped 1 previous similar message [ 5626.979968] Lustre: lustre-MDT0001: new disk, initializing [ 5627.165443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5627.277627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5627.309404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5633.914271] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5647.218323] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.050460] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.967521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.476498] Lustre: lustre-OST0000: new disk, initializing [ 5666.479732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5666.487345] Lustre: 161713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5666.638336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.794602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5675.817853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5676.094924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5676.144750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5688.486708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5694.415504] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5697.482452] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:31:44 (1787466704) [ 5990.880474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5990.906546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.958606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.512265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.514829] Lustre: Skipped 5 previous similar messages [ 5996.954976] Lustre: server umount lustre-MDT0000 complete [ 6001.191745] LustreError: 159736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467009 with bad export cookie 10116127526046449534 [ 6001.196737] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.206249] LustreError: 159736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6001.632201] LustreError: 161516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.632637] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.657210] Lustre: Skipped 2 previous similar messages [ 6001.660536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6006.752504] LustreError: 159750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.752559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6007.781031] Lustre: server umount lustre-MDT0001 complete [ 6028.255212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.593429] Lustre: server umount lustre-OST0000 complete [ 6041.099204] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6054.620291] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6101.275237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.560387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6101.594823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6101.680595] Lustre: lustre-MDT0000: new disk, initializing [ 6101.829493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.859401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6108.375609] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6122.401418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.577420] Lustre: 166387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6122.709857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6122.720221] Lustre: Skipped 1 previous similar message [ 6122.924067] Lustre: lustre-MDT0001: new disk, initializing [ 6123.120497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.181047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6123.203709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6129.801710] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6142.849244] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6149.728635] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6158.326349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6158.621819] Lustre: lustre-OST0000: new disk, initializing [ 6158.626129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6158.635086] Lustre: 168358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6158.733165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.501206] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6168.796401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6168.828328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6168.993643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6178.796130] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6186.047912] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6186.475233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6193.697975] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6194.071376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6199.777534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6199.791503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.797078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.815722] Lustre: Skipped 1 previous similar message [ 6199.863481] Lustre: Skipped 1 previous similar message [ 6203.262965] Lustre: server umount lustre-OST0000 complete [ 6210.016799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6210.021645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.027744] LustreError: Skipped 1 previous similar message [ 6210.030908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6210.053687] Lustre: Skipped 1 previous similar message [ 6214.458397] Lustre: server umount lustre-MDT0000 complete [ 6215.148541] LustreError: 166398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.195694] LustreError: 166398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6219.775730] LustreError: 166377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467227 with bad export cookie 10116127526046608371 [ 6219.787983] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6219.797352] LustreError: 166377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6220.429412] Lustre: server umount lustre-MDT0001 complete [ 6231.591536] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:40:39 (1787467239) [ 6255.730654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6272.690526] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6323.831326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.253614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6324.288137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6324.392598] Lustre: lustre-MDT0000: new disk, initializing [ 6324.522439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.550770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6330.476873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6343.642433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.879425] Lustre: 173139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6344.311805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6350.778687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6362.809180] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6370.696771] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.024901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.458947] Lustre: lustre-OST0000: new disk, initializing [ 6380.462889] Lustre: Skipped 1 previous similar message [ 6380.468281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6380.475109] Lustre: Skipped 2 previous similar messages [ 6380.482562] Lustre: 175120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6388.396897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6388.409316] Lustre: Skipped 1 previous similar message [ 6388.421358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6388.609124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6389.683788] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6402.750351] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.304521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.310487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.323057] LustreError: Skipped 1 previous similar message [ 6414.324941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.325133] Lustre: Skipped 1 previous similar message [ 6414.367752] Lustre: Skipped 1 previous similar message [ 6416.512517] Lustre: server umount lustre-OST0000 complete [ 6424.544509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.564889] Lustre: Skipped 1 previous similar message [ 6436.319182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6436.805499] Lustre: server umount lustre-MDT0000 complete [ 6439.914196] LustreError: 173152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.941183] LustreError: 173152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6441.207442] LustreError: 173134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467449 with bad export cookie 10116127526046609253 [ 6441.215398] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.216402] LustreError: 173134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6441.787767] Lustre: server umount lustre-MDT0001 complete [ 6453.111523] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:44:20 (1787467460) [ 6474.773594] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6488.443108] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6538.230850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.536676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6538.601436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6538.738681] Lustre: lustre-MDT0000: new disk, initializing [ 6538.880362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.889329] Lustre: Skipped 2 previous similar messages [ 6538.939434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6545.558512] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6559.633181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.931627] Lustre: 179449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6560.463170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6566.239148] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6579.956660] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6588.115853] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6597.129402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.565947] Lustre: 181428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6603.436858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6603.657136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6606.026865] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6620.067554] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6634.473741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6634.476467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.518966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6634.526141] Lustre: Skipped 8 previous similar messages [ 6635.910795] Lustre: server umount lustre-OST0000 complete [ 6642.660889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6642.687983] LustreError: Skipped 1 previous similar message [ 6647.024729] Lustre: server umount lustre-MDT0000 complete [ 6649.826230] LustreError: 179459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.860584] LustreError: 179459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6651.997879] LustreError: 179441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467660 with bad export cookie 10116127526046610338 [ 6652.007732] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.016477] LustreError: 179441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.058767] LustreError: 155697:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 6652.588297] Lustre: server umount lustre-MDT0001 complete [ 6672.950346] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6676.796478] Key type lgssc unregistered [ 6677.333115] LNet: 183315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6677.369154] LNetError: 183315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6677.391975] LNet: Removed LNI 192.168.202.102@tcp [ 6678.717719] Key type .llcrypt unregistered [ 6678.721300] Key type ._llcrypt unregistered [ 6713.150543] Key type ._llcrypt registered [ 6713.152023] Key type .llcrypt registered [ 6713.267168] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6715.138474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.363798] alg: No test for adler32 (adler32-zlib) [ 6716.480814] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6716.918518] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6718.759237] Key type lgssc registered [ 6720.455337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6733.757303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6733.795440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.820359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.852907] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6745.066532] LustreError: 184736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.127358] LustreError: 184736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6750.194790] LustreError: 184737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.301939] LustreError: 184736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.363790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.044361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6764.916432] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6776.521234] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6784.943500] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6794.261739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6794.829941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.331531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6805.129229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6814.356689] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6836.200901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.209918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.218488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.240792] Lustre: Skipped 1 previous similar message [ 6841.315134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.331385] Lustre: Skipped 1 previous similar message [ 6842.501334] Lustre: server umount lustre-OST0000 complete [ 6848.994616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6849.003611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6849.024254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6849.032566] Lustre: Skipped 1 previous similar message [ 6851.559892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.561277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.989988] Lustre: server umount lustre-MDT0000 complete [ 6856.677968] LustreError: 185465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.980414] LustreError: 184716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467868 with bad export cookie 4406576412738885639 [ 6859.984802] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.005905] LustreError: 184716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6860.685316] Lustre: server umount lustre-MDT0001 complete [ 6875.110978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6881.522378] Key type lgssc unregistered [ 6882.121059] LNet: 188297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6882.134415] LNetError: 188297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6883.175603] LNet: Removed LNI 192.168.202.102@tcp [ 6884.820656] Key type .llcrypt unregistered [ 6884.824239] Key type ._llcrypt unregistered [ 6922.014304] Key type ._llcrypt registered [ 6922.016468] Key type .llcrypt registered [ 6922.106191] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6924.311700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6924.393942] alg: No test for adler32 (adler32-zlib) [ 6925.799870] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6926.132256] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6927.919393] Key type lgssc registered [ 6929.530073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6941.122295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6941.146271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.030552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.350735] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6953.440573] LustreError: 189718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6953.462805] LustreError: 189718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6958.599528] LustreError: 189717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.686291] LustreError: 189718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.712466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.800875] LustreError: 189717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.422705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.659293] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6987.213918] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.563882] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7005.204445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7005.848235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.093450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7014.312484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7025.856288] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.944592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.950664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.963022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7039.972842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7039.977978] Lustre: Skipped 1 previous similar message [ 7039.979146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7039.991116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.752547] Lustre: server umount lustre-OST0000 complete [ 7054.040470] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa08d04e7b000.ost_server_uuid 50 [ 7056.377717] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7075.122088] Lustre: server umount lustre-MDT0000 complete [ 7075.816617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.847378] LustreError: 189714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.884912] LustreError: 189714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7080.705506] LustreError: 189698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468088 with bad export cookie 7789520294219456168 [ 7080.714614] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.091802] Lustre: server umount lustre-MDT0001 complete [ 7092.121739] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:55:00 (1787468100) [ 7102.056811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.930427] LustreError: 193640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.960445] LustreError: 193640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7103.028675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7109.377501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7120.249841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.766289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7126.743885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7137.134141] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7145.797824] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.811608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7156.552478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.954441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7173.978219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7174.163620] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.368822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.644403] Lustre: lustre-OST0001: new disk, initializing [ 7182.647786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7182.664340] Lustre: 196483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7182.768122] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7190.730746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7190.749387] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7190.972960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7192.542242] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7205.587458] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.957282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.971097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.001728] Lustre: Skipped 1 previous similar message [ 7210.011240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7210.031198] Lustre: Skipped 1 previous similar message [ 7215.952456] Lustre: server umount lustre-OST0000 complete [ 7216.632399] LustreError: 196781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.676367] LustreError: 196781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7228.940411] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7229.107939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7236.137623] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7236.559521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7252.453456] LustreError: 195462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.484747] LustreError: 195462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 7254.898740] Lustre: server umount lustre-OST0001 complete [ 7262.702491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.711438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.726404] Lustre: Skipped 2 previous similar messages [ 7262.755356] Lustre: Skipped 4 previous similar messages [ 7267.810648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.813274] Lustre: Skipped 1 previous similar message [ 7270.994805] Lustre: server umount lustre-MDT0000 complete [ 7273.956671] Lustre: 189027:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787468265/real 1787468265] req@000000008850263c x1874295976476032/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787468281 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7273.983071] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.390461] LustreError: 193623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468283 with bad export cookie 7789520294219457239 [ 7275.393946] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.402406] LustreError: 193623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7275.747449] Lustre: server umount lustre-MDT0001 complete [ 7305.512301] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:58:32 (1787468312) [ 7313.813273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.334756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7314.398087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7314.780535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.627078] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7331.345041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.606643] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7331.706040] Lustre: 199923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7338.393404] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7347.541177] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7355.225982] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7363.654124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7363.868577] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7372.022526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7372.076587] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7381.621836] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.642809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.729340] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7392.058487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7392.072934] Lustre: Skipped 2 previous similar messages [ 7400.216666] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7400.932218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7411.014872] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7416.309187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.329819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7416.339171] Lustre: Skipped 1 previous similar message [ 7416.360630] Lustre: Skipped 1 previous similar message [ 7429.088164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7429.253267] Lustre: server umount lustre-OST0000 complete [ 7431.653674] LustreError: 201774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.680519] LustreError: 201774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7447.009403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7447.022203] LustreError: Skipped 1 previous similar message [ 7447.034538] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7447.046414] Lustre: Skipped 8 previous similar messages [ 7453.361549] Lustre: server umount lustre-OST0001 complete [ 7460.321922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7460.339830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.369981] Lustre: Skipped 2 previous similar messages [ 7464.901803] Lustre: server umount lustre-MDT0000 complete [ 7469.964855] LustreError: 199915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468478 with bad export cookie 7789520294219458646 [ 7469.970553] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7469.974106] LustreError: 199915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7470.198397] Lustre: server umount lustre-MDT0001 complete [ 7501.286613] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:01:48 (1787468508) [ 7522.263332] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7533.671554] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7591.076811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.547754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7591.697739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7591.837964] Lustre: lustre-MDT0000: new disk, initializing [ 7591.916970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7591.952919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7598.150691] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7612.113433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.246888] Lustre: 208050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7612.367169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7612.382767] Lustre: Skipped 1 previous similar message [ 7612.557934] Lustre: lustre-MDT0001: new disk, initializing [ 7612.797703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7612.828945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7620.465327] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7633.766871] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7642.361244] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7649.965971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.590560] Lustre: lustre-OST0000: new disk, initializing [ 7650.592492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7650.604520] Lustre: 210020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7657.719170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7657.764807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7657.828225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7658.588501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7673.247831] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.157824] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7680.333378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7687.395140] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7687.980961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7693.803906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.824410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.836000] Lustre: Skipped 1 previous similar message [ 7693.841889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.851162] Lustre: Skipped 6 previous similar messages [ 7706.079663] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7706.373536] Lustre: server umount lustre-OST0000 complete [ 7719.058747] Lustre: server umount lustre-MDT0000 complete [ 7719.401478] LustreError: 208057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.424493] LustreError: 208057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7724.032330] LustreError: 208846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468732 with bad export cookie 7789520294219460431 [ 7724.039378] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7724.046328] LustreError: 208846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7724.506751] Lustre: server umount lustre-MDT0001 complete [ 7735.117557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.241793] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7750.995604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.619075] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7766.280462] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7774.451309] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7783.727894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.393345] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7802.651293] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.335170] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7809.728468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7815.831393] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7816.138940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7821.293398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.305483] Lustre: Skipped 6 previous similar messages [ 7825.052216] Lustre: server umount lustre-OST0000 complete [ 7832.352531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7838.907222] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7869.782650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.995313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.001787] Lustre: Skipped 5 previous similar messages [ 7880.708837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7886.449181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7886.492233] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7886.492326] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7892.603820] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7894.941042] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7902.176903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.190651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.200096] Lustre: Skipped 5 previous similar messages [ 7906.563726] Lustre: server umount lustre-OST0000 complete [ 7918.314625] Lustre: server umount lustre-MDT0000 complete [ 7925.986674] LustreError: 212226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468934 with bad export cookie 7789520294219461320 [ 7926.009066] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7926.036951] LustreError: 212226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7927.117029] Lustre: server umount lustre-MDT0001 complete [ 7942.433611] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:09:09 (1787468949) [ 7953.997842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.727722] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7973.404241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.319326] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7992.654839] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8000.538956] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8009.830297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8018.561573] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8027.527240] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8035.899875] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8036.425513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8044.119445] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8044.849636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8056.409349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.676574] Lustre: lustre-OST0001: new disk, initializing [ 8056.679617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8056.684594] Lustre: 221602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8063.748334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8063.767032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8064.117985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8068.010983] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8082.169975] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8089.894866] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8090.289637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8097.302937] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8097.545173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8104.945077] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.953307] LustreError: Skipped 2 previous similar messages [ 8104.963145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8104.966296] Lustre: Skipped 7 previous similar messages [ 8106.312256] Lustre: server umount lustre-OST0001 complete [ 8110.074280] LustreError: 220050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.112639] LustreError: 220050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8116.343701] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8116.636276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8123.204279] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8123.609557] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8156.894297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.288859] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8162.322467] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8162.325832] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8162.347545] Lustre: Skipped 2 previous similar messages [ 8166.110874] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8177.331827] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8180.134219] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8187.903558] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.943306] Lustre: Skipped 6 previous similar messages [ 8189.477187] Lustre: server umount lustre-OST0001 complete [ 8213.474918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.250912] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8225.417382] Lustre: server umount lustre-OST0000 complete [ 8238.923814] Lustre: server umount lustre-MDT0000 complete [ 8243.493593] LustreError: 218250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469251 with bad export cookie 7789520294219462286 [ 8243.497026] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8243.509050] LustreError: 218250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8249.955917] Lustre: server umount lustre-MDT0001 complete [ 8282.541266] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:14:49 (1787469289) [ 8290.154766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.665664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8290.704470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8295.674646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8305.024947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.123420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8305.156731] Lustre: 226854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8312.649358] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8325.770516] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8334.364537] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8344.066539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8344.422495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8353.860727] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8365.066566] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8377.881712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.919873] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8385.797604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8395.353745] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8397.365638] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8402.302737] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8402.502573] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8407.760697] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8407.961691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8416.815916] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa08d09bad800.ost_server_uuid 50 [ 8420.050970] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa08d09bad800.ost_server_uuid in IDLE state after 0 sec [ 8421.840499] Lustre: Permanently deactivating lustre-OST0001 [ 8421.913577] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8429.573675] Lustre: Permanently reactivating lustre-OST0001 [ 8429.639836] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8429.654372] Lustre: Skipped 2 previous similar messages [ 8437.217298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.230017] LustreError: Skipped 2 previous similar messages [ 8437.231022] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8437.238366] Lustre: Skipped 11 previous similar messages [ 8441.707778] Lustre: server umount lustre-OST0001 complete [ 8451.711352] Lustre: server umount lustre-OST0000 complete [ 8462.035432] Lustre: server umount lustre-MDT0000 complete [ 8468.177042] LustreError: 226846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469476 with bad export cookie 7789520294219463574 [ 8468.199996] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8468.200849] LustreError: 226846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8468.887195] Lustre: server umount lustre-MDT0001 complete [ 8505.927968] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:18:33 (1787469513) [ 8516.610852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.102295] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8517.169425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8517.176458] Lustre: Skipped 2 previous similar messages [ 8517.455507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.460860] Lustre: Skipped 9 previous similar messages [ 8522.867400] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8533.727241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8533.895688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8533.915745] Lustre: 233562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8539.678956] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8549.375555] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8556.346062] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8564.575069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8564.840403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8564.883458] Lustre: Permanently deactivating lustre-OST0000 [ 8564.907800] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8572.867732] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8572.940981] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8582.969614] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8593.337543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.485251] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8603.608417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8604.960501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8613.995346] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8616.750875] Lustre: Permanently reactivating lustre-OST0000 [ 8616.776792] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8616.791648] Lustre: Skipped 2 previous similar messages [ 8623.728947] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 8623.745471] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp 000000001266056b, cur 1787469631 deadline 1787469611 last 1787469581 [ 8623.844698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8639.460645] LustreError: 236745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8639.488867] LustreError: 236745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 51 previous similar messages [ 8639.535474] Lustre: server umount lustre-OST0001 complete [ 8650.475389] Lustre: server umount lustre-OST0000 complete [ 8662.116610] Lustre: server umount lustre-MDT0000 complete [ 8666.193691] LustreError: 233553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469674 with bad export cookie 7789520294219465247 [ 8666.196642] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.202803] LustreError: 233553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8666.602852] Lustre: server umount lustre-MDT0001 complete [ 8679.510427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8682.887947] Key type lgssc unregistered [ 8683.171619] LNet: 238443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8683.181477] LNetError: 238443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8683.199755] LNet: Removed LNI 192.168.202.102@tcp [ 8684.277281] Key type .llcrypt unregistered [ 8684.278949] Key type ._llcrypt unregistered [ 8711.806855] Key type ._llcrypt registered [ 8711.811294] Key type .llcrypt registered [ 8711.959317] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8713.251707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8713.497670] alg: No test for adler32 (adler32-zlib) [ 8714.814371] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 8715.116548] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8716.895427] Key type lgssc registered [ 8718.311290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8730.861996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8730.897826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.718781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.753843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8742.880671] LustreError: 239913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8742.915043] LustreError: 239913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8748.009990] LustreError: 239914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.726692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8749.873978] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8749.923533] Lustre: 239901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8749.938888] Lustre: Permanently deactivating lustre-MDT0001 [ 8749.965367] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8750.269901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8755.333344] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8756.234285] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8765.364904] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8772.439125] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8781.506911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8782.407820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.247300] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8795.696720] LustreError: 241758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.755550] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8800.759571] LustreError: 241759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.187490] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8802.926732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8807.920879] LustreError: 241759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.948755] LustreError: 241759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8809.986320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.240915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8815.657548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8815.672272] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8818.182288] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8828.189566] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8831.474711] Lustre: Permanently reactivating lustre-MDT0001 [ 8831.488314] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8831.491307] Lustre: Skipped 2 previous similar messages [ 8834.758817] Lustre: lustre-MDT0001: Client 8418eed3-2062-48cc-ae13-ab78d2cb7aa6 (at 192.168.202.2@tcp) reconnecting [ 8839.910945] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.955100] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8839.968712] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8839.993684] LustreError: 243627:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 8839.996332] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8840.013612] LustreError: 243627:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 8840.013966] LustreError: 243627:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -108 [ 8840.014115] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 8840.014115] [ 8840.015941] LustreError: 239904:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 8841.659659] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8843.780733] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8851.279938] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8851.632497] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8857.124409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8857.332445] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8860.534140] Lustre: Permanently deactivating lustre-MDT0001 [ 8860.581768] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8860.590821] Lustre: Skipped 2 previous similar messages [ 8867.969133] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8877.025563] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.034032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8877.054258] Lustre: Skipped 1 previous similar message [ 8881.226844] Lustre: server umount lustre-MDT0000 complete [ 8882.146750] LustreError: 241586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.184053] LustreError: 241586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8885.587583] LustreError: 239893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469893 with bad export cookie 5005002483758695486 [ 8885.605383] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8885.612971] LustreError: 239893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8885.889100] Lustre: server umount lustre-MDT0001 complete [ 8892.603165] Lustre: server umount lustre-OST0000 complete [ 8907.538067] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:25:14 (1787469914) [ 8920.687279] Lustre: server umount lustre-OST0001 complete [ 8928.555405] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8941.912640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8986.894891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.092665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8987.096648] Lustre: Skipped 2 previous similar messages [ 8987.116064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8987.186310] Lustre: lustre-MDT0000: new disk, initializing [ 8987.260169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.281730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8992.533536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9005.409227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.566589] Lustre: 248180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9005.626408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9005.644768] Lustre: Skipped 1 previous similar message [ 9005.722681] Lustre: lustre-MDT0001: new disk, initializing [ 9005.844267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9005.916348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9005.929034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9011.886604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9022.837361] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9030.431827] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9039.672945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.014989] Lustre: lustre-OST0000: new disk, initializing [ 9040.021939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9040.030462] Lustre: 250150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9040.136880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.140926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9047.156089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9047.297768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9047.975542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9059.982770] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9062.707985] LustreError: 248187:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9072.736213] LustreError: 248187:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9072.765148] LustreError: 248188:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9072.881202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9073.093949] Lustre: lustre-OST0001: new disk, initializing [ 9073.098794] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9073.105747] Lustre: 251266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9073.197734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9080.523714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9081.975450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9081.987396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9082.079866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9082.815144] LustreError: 248188:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9091.061088] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9093.065113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9097.699708] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9097.712668] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.714535] LustreError: Skipped 1 previous similar message [ 9097.723455] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9097.739960] Lustre: Skipped 3 previous similar messages [ 9097.759961] Lustre: Skipped 4 previous similar messages [ 9101.598037] Lustre: server umount lustre-OST0001 complete [ 9102.820502] LustreError: 250458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9102.852462] LustreError: 250458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9123.305857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.315927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9123.323039] Lustre: Skipped 1 previous similar message [ 9123.331992] Lustre: Skipped 1 previous similar message [ 9125.836321] Lustre: server umount lustre-OST0000 complete [ 9133.537666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9133.546276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9133.551710] Lustre: Skipped 1 previous similar message [ 9133.558449] Lustre: Skipped 1 previous similar message [ 9136.790746] Lustre: server umount lustre-MDT0000 complete [ 9138.659160] LustreError: 252123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.687086] LustreError: 252123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9141.053299] LustreError: 248976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470149 with bad export cookie 5005002483758698300 [ 9141.063801] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9141.072678] LustreError: 248976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9147.499736] Lustre: server umount lustre-MDT0001 complete [ 9160.337423] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9163.494250] Key type lgssc unregistered [ 9163.817046] LNet: 253304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9163.829367] LNetError: 253304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9163.852666] LNet: Removed LNI 192.168.202.102@tcp [ 9164.979537] Key type .llcrypt unregistered [ 9164.982131] Key type ._llcrypt unregistered [ 9210.817365] Key type ._llcrypt registered [ 9210.819144] Key type .llcrypt registered [ 9210.897699] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9211.889806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9211.930873] alg: No test for adler32 (adler32-zlib) [ 9213.112990] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9213.321863] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9215.015365] Key type lgssc registered [ 9216.134535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9224.108630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9224.135725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.434181] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9225.452785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9225.600760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.912168] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9240.315252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.458871] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9240.508991] Lustre: 255513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9240.972353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9246.246721] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9256.028683] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9263.361348] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9271.836283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9272.121343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9272.425627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9280.787426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 9280.793654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9290.285612] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9347.557897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9347.560416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.563044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9347.578377] LustreError: Skipped 1 previous similar message [ 9347.619067] Lustre: Skipped 1 previous similar message [ 9351.042721] Lustre: server umount lustre-OST0000 complete [ 9356.380715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 9372.126937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.542352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.032989] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9387.881518] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9387.905599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9387.917827] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9387.949581] Lustre: Skipped 1 previous similar message [ 9389.155907] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9434.080578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9434.103678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.121679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.145963] Lustre: Skipped 1 previous similar message [ 9434.157369] Lustre: Skipped 1 previous similar message [ 9436.966168] Lustre: server umount lustre-OST0000 complete [ 9444.325845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.354698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.369203] Lustre: Skipped 1 previous similar message [ 9444.395719] Lustre: Skipped 2 previous similar messages [ 9449.114808] Lustre: server umount lustre-MDT0000 complete [ 9453.731104] LustreError: 255505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470461 with bad export cookie 4676820539091054448 [ 9453.744461] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9453.746599] LustreError: 255505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9454.418189] Lustre: server umount lustre-MDT0001 complete [ 9467.564254] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9470.836460] Key type lgssc unregistered [ 9471.192064] LNet: 260946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9471.197635] LNetError: 260946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9471.220931] LNet: Removed LNI 192.168.202.102@tcp [ 9472.390909] Key type .llcrypt unregistered [ 9472.393772] Key type ._llcrypt unregistered [ 9498.817614] Key type ._llcrypt registered [ 9498.818833] Key type .llcrypt registered [ 9498.921121] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9500.276105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9500.436731] alg: No test for adler32 (adler32-zlib) [ 9501.568441] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9501.831761] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9503.559224] Key type lgssc registered [ 9505.064147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9514.543939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9514.573654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.130515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.590974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9526.245828] LustreError: 262366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.273926] LustreError: 262366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9531.181850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.367841] LustreError: 262365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.563154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9536.704250] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9546.046616] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9553.334813] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9561.404846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.650814] LustreError: 264186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.782200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9567.239369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 9569.915651] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9579.390617] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9607.154111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.156113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9607.170299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9610.364200] Lustre: server umount lustre-OST0000 complete [ 9617.377409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.388956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.401541] Lustre: Skipped 2 previous similar messages [ 9617.409278] Lustre: Skipped 1 previous similar message [ 9620.958338] Lustre: server umount lustre-MDT0000 complete [ 9622.499409] LustreError: 262366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.526751] LustreError: 262366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9625.305908] LustreError: 262345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470633 with bad export cookie 10910819157879628234 [ 9625.315249] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.330412] LustreError: 262345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9626.016361] Lustre: server umount lustre-MDT0001 complete [ 9641.543908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9644.825929] Key type lgssc unregistered [ 9645.395653] LNet: 266675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9645.414879] LNetError: 266675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9645.501146] LNet: Removed LNI 192.168.202.102@tcp [ 9646.537289] Key type .llcrypt unregistered [ 9646.540310] Key type ._llcrypt unregistered [ 9670.472749] Key type ._llcrypt registered [ 9670.475569] Key type .llcrypt registered [ 9670.602427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9672.698859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9672.735909] alg: No test for adler32 (adler32-zlib) [ 9673.932476] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9674.312825] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9676.119361] Key type lgssc registered [ 9677.630488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9687.441965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9687.460390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9689.189025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9694.188594] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9699.297488] LustreError: 267899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9699.348517] LustreError: 267899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9704.423092] LustreError: 267900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9707.656273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.101430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9713.802507] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9722.031095] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9730.345278] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9738.981224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9739.500835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.784273] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9756.653841] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9758.982954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9774.406155] Lustre: server umount lustre-OST0000 complete [ 9779.684822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9779.688604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.722501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9784.801248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9784.804407] Lustre: Skipped 2 previous similar messages [ 9789.927750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9789.932281] Lustre: Skipped 1 previous similar message [ 9790.023501] Lustre: server umount lustre-MDT0000 complete [ 9790.765225] Lustre: 267212:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787470782/real 1787470782] req@000000007e5d98bf x1874298857880192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787470798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9790.789811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.798539] Lustre: Skipped 1 previous similar message [ 9790.807540] LustreError: 267900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.837419] LustreError: 267900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9794.260691] LustreError: 267880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470802 with bad export cookie 16414463996972540691 [ 9794.281054] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.624703] Lustre: server umount lustre-MDT0001 complete [ 9807.306564] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9810.316182] Key type lgssc unregistered [ 9810.587708] LNet: 271585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9810.593444] LNetError: 271585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9810.617047] LNet: Removed LNI 192.168.202.102@tcp [ 9811.523083] Key type .llcrypt unregistered [ 9811.525159] Key type ._llcrypt unregistered [ 9840.327243] Key type ._llcrypt registered [ 9840.328692] Key type .llcrypt registered [ 9840.512444] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9842.038475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9842.393449] alg: No test for adler32 (adler32-zlib) [ 9843.564523] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9843.844040] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9845.591248] Key type lgssc registered [ 9847.132980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9857.242890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9857.274805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.080710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9865.443828] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9869.280297] LustreError: 273003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.300621] LustreError: 273003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9874.417859] LustreError: 273004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.818282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.252925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.905596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9892.067872] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9899.420653] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9908.392616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9909.045655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.735796] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9927.022474] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9929.542429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9962.471237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9962.475491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.496987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9965.536884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.538544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9967.695180] Lustre: server umount lustre-OST0000 complete [ 9973.739716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9973.746826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9973.770507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9973.791945] Lustre: Skipped 1 previous similar message [ 9975.776395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.778900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.239745] Lustre: server umount lustre-MDT0000 complete [ 9980.897278] LustreError: 274269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9980.918235] LustreError: 274269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9983.917642] LustreError: 272986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470991 with bad export cookie 6735506810107829236 [ 9983.924194] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9983.941547] LustreError: 272986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9984.374833] Lustre: server umount lustre-MDT0001 complete [ 9998.683176] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10002.421338] Key type lgssc unregistered [10002.720565] LNet: 277316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10002.738320] LNetError: 277316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10003.814340] LNet: Removed LNI 192.168.202.102@tcp [10005.246398] Key type .llcrypt unregistered [10005.251270] Key type ._llcrypt unregistered [10025.605440] Key type ._llcrypt registered [10025.608912] Key type .llcrypt registered [10025.727812] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10026.851867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10026.878993] alg: No test for adler32 (adler32-zlib) [10028.187638] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [10028.423921] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10030.207214] Key type lgssc registered [10031.798411] Lustre: Echo OBD driver; http://www.lustre.org/ [10041.433219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10041.453863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.284978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.771487] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10053.604846] LustreError: 278555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.645218] LustreError: 278555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10058.724162] LustreError: 278557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.369862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.687194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10066.714405] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10076.622954] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10084.996866] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10093.925973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.470364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10102.782846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10113.931087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [10114.224651] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10129.377277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10129.390420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.395437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10133.984128] Lustre: server umount lustre-OST0000 complete [10139.617816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.630967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.631453] Lustre: Skipped 2 previous similar messages [10139.659728] Lustre: Skipped 2 previous similar messages [10144.736245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.748086] Lustre: Skipped 1 previous similar message [10145.242509] Lustre: server umount lustre-MDT0000 complete [10149.864652] LustreError: 278555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.885659] LustreError: 278555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10149.947353] LustreError: 278521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471158 with bad export cookie 16150616651859605786 [10149.970287] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10150.394259] Lustre: server umount lustre-MDT0001 complete [10164.697774] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10168.192122] Key type lgssc unregistered [10168.607842] LNet: 282264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10168.615710] LNetError: 282264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10169.638445] LNet: Removed LNI 192.168.202.102@tcp [10170.996496] Key type .llcrypt unregistered [10171.002529] Key type ._llcrypt unregistered [10185.281661] Key type ._llcrypt registered [10185.283854] Key type .llcrypt registered [10185.415764] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [10207.991884] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10224.269108] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10225.001356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10225.179915] alg: No test for adler32 (adler32-zlib) [10226.112975] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [10226.337704] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10228.031260] Key type lgssc registered [10229.509089] Lustre: Echo OBD driver; http://www.lustre.org/ [10275.397959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10275.442827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.997835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10277.032129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10277.107129] Lustre: lustre-MDT0000: new disk, initializing [10277.174637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.194963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10283.005860] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10296.142940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10296.308726] Lustre: 285822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10296.395094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10296.399571] Lustre: Skipped 1 previous similar message [10296.569410] Lustre: lustre-MDT0001: new disk, initializing [10296.729479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10296.785034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10296.798367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10302.808392] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10314.197793] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10321.312492] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10329.818687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10330.147656] Lustre: lustre-OST0000: new disk, initializing [10330.150892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10330.156596] Lustre: 287792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10330.277950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10337.188234] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10337.443275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10337.465242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10337.614060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10348.602797] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10354.893564] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10355.158203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10360.402898] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10360.667700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10363.391932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.401760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10363.427873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.440762] LustreError: Skipped 1 previous similar message [10363.453676] Lustre: Skipped 1 previous similar message [10368.483302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10368.489801] Lustre: Skipped 1 previous similar message [10369.412322] Lustre: server umount lustre-OST0000 complete [10378.723441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.749625] Lustre: Skipped 1 previous similar message [10378.756880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10378.763562] Lustre: Skipped 1 previous similar message [10380.888497] Lustre: server umount lustre-MDT0000 complete [10385.991729] LustreError: 285813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471394 with bad export cookie 16654599532778620029 [10385.999713] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10386.001882] LustreError: 285813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10386.355554] Lustre: server umount lustre-MDT0001 complete [10397.604842] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:50:05 (1787471405) [10406.060275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.526857] LustreError: 290205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.569099] LustreError: 290205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10406.684708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10412.011363] LustreError: 290206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.121841] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10416.101247] LustreError: 290205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.227205] LustreError: 290206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.308284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.735442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10429.443895] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10438.257402] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10444.112941] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.905917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10452.384256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10460.432889] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10470.616401] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10488.291373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10488.297205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.321079] Lustre: Skipped 1 previous similar message [10488.336159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10490.075867] Lustre: server umount lustre-OST0000 complete [10496.483506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.487483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.494461] Lustre: Skipped 1 previous similar message [10504.778785] Lustre: server umount lustre-MDT0000 complete [10506.725679] LustreError: 290206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.742868] LustreError: 290206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10509.999366] LustreError: 292049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471518 with bad export cookie 16654599532778620904 [10510.009748] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10510.019423] LustreError: 292049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10510.471705] Lustre: server umount lustre-MDT0001 complete [10523.909570] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10527.184745] Key type lgssc unregistered [10527.518744] LNet: 293766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10527.528045] LNetError: 293766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10528.550271] LNet: Removed LNI 192.168.202.102@tcp [10529.553170] Key type .llcrypt unregistered [10529.554767] Key type ._llcrypt unregistered [10563.121944] Key type ._llcrypt registered [10563.124926] Key type .llcrypt registered [10563.268572] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10579.658222] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10581.118774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10581.143591] alg: No test for adler32 (adler32-zlib) [10582.425587] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [10582.865901] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10584.664354] Key type lgssc registered [10586.460252] Lustre: Echo OBD driver; http://www.lustre.org/ [10648.480769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10648.531889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.921523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10649.974202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10650.087564] Lustre: lustre-MDT0000: new disk, initializing [10650.220431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10650.281496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10655.740675] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10667.455934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10667.606202] Lustre: 297613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10667.667475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10667.670535] Lustre: Skipped 1 previous similar message [10667.765300] Lustre: lustre-MDT0001: new disk, initializing [10667.900346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10667.954385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10667.969907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10673.432871] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10685.057621] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10691.885791] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10699.993899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10700.598526] Lustre: lustre-OST2710: new disk, initializing [10700.603563] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [10700.615750] Lustre: 299582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10700.769562] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10708.463385] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10710.101463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10710.117923] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10710.273870] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [10721.165954] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10730.586624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.716116] Lustre: lustre-OST03e8: new disk, initializing [10730.718074] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10730.720966] Lustre: 300650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10730.794915] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10738.881122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10738.906183] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10739.113265] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10739.595582] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10749.689734] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10757.983508] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10758.351566] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10764.873200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10765.330555] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10772.350126] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10772.611704] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10780.045940] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10780.410057] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10790.387914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.399693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10790.425708] Lustre: Skipped 1 previous similar message [10792.594068] Lustre: server umount lustre-MDT0000 complete [10796.006182] LustreError: 297606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471804 with bad export cookie 15070579434216492531 [10796.011245] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10796.016316] LustreError: 297606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10796.759485] Lustre: server umount lustre-MDT0001 complete [10812.272933] Lustre: server umount lustre-OST2710 complete [10815.967322] Lustre: 295408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787471808/real 1787471808] req@000000005d0d4d77 x1874299810036096/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787471824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10816.005080] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10816.023273] Lustre: Skipped 2 previous similar messages [10817.103923] Lustre: server umount lustre-OST03e8 complete [10838.456700] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10847.524690] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10888.419767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.716392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10888.752475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10888.860618] Lustre: lustre-MDT0000: new disk, initializing [10888.978652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10889.035977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10893.578118] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10904.320887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.452369] Lustre: 305906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10904.495673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10904.499736] Lustre: Skipped 1 previous similar message [10904.584460] Lustre: lustre-MDT0001: new disk, initializing [10904.670954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10904.705882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10904.722810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10910.246741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10920.385820] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10926.002806] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10933.765474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10934.131668] Lustre: lustre-OST0000: new disk, initializing [10934.136310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10934.142956] Lustre: 307872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10934.241187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10941.581915] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10942.016404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10942.027374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10942.222238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10951.181725] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10955.627247] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10955.833878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10961.047714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10961.221425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10968.033735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10968.058163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.085565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10968.095656] Lustre: Skipped 4 previous similar messages [10969.555284] Lustre: server umount lustre-OST0000 complete [10976.224970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10976.232702] LustreError: Skipped 1 previous similar message [10976.236376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.243137] Lustre: Skipped 1 previous similar message [10976.248496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.274861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.286402] Lustre: Skipped 1 previous similar message [10979.957216] Lustre: server umount lustre-MDT0000 complete [10983.396307] LustreError: 305911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.415432] LustreError: 305911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10984.139974] LustreError: 305897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471992 with bad export cookie 15070579434216494225 [10984.146962] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10984.149066] LustreError: 305897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10984.601281] Lustre: server umount lustre-MDT0001 complete [10995.871801] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:00:03 (1787472003) [10997.963901] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10999.767939] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:00:07 (1787472007) [11003.885950] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11039.236753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.735818] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11039.808649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11040.162486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11045.962941] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11056.204366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11056.386541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11056.450450] Lustre: 311730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11061.991786] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11085.549727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11085.851869] Lustre: Denying initial registration attempt from nid 192.168.202.102@tcp, specified as failover [11085.871214] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [11085.880478] LustreError: 313195:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [11085.885349] LustreError: 313195:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11085.890040] LustreError: 313195:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11086.065209] Lustre: server umount lustre-OST0000 complete [11086.071495] LustreError: 313195:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [11098.081353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11098.082175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11098.096887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11098.115342] Lustre: Skipped 2 previous similar messages [11098.126167] Lustre: Skipped 1 previous similar message [11108.319696] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11108.322180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11108.348302] Lustre: Skipped 3 previous similar messages [11108.505796] Lustre: server umount lustre-MDT0000 complete [11112.845011] LustreError: 311721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472120 with bad export cookie 15070579434216495107 [11112.861295] LustreError: 311721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11112.869843] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.497930] Lustre: server umount lustre-MDT0001 complete [11128.638994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11132.529776] Key type lgssc unregistered [11133.179883] LNet: 314321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11133.203607] LNetError: 314321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11134.250850] LNet: Removed LNI 192.168.202.102@tcp [11135.410310] Key type .llcrypt unregistered [11135.418907] Key type ._llcrypt unregistered [11150.198811] Key type ._llcrypt registered [11150.201409] Key type .llcrypt registered [11150.319306] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11151.577445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11151.587333] alg: No test for adler32 (adler32-zlib) [11152.664583] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [11153.107940] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11154.871236] Key type lgssc registered [11156.103687] Lustre: Echo OBD driver; http://www.lustre.org/ [11211.744625] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11222.235201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11222.257854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.521853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11223.549350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11223.679198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11229.004910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11240.357814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.534104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11240.579840] Lustre: 317090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11241.008744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11247.108331] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11256.883297] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11263.255411] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11284.207186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11284.530602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11284.841464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.722663] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11301.896756] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11306.468776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11306.478323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.499147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.994958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11307.008503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.034645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11310.762499] Lustre: server umount lustre-OST0000 complete [11317.223741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.243977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.255849] Lustre: Skipped 1 previous similar message [11321.436961] Lustre: server umount lustre-MDT0000 complete [11326.679677] LustreError: 317082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472334 with bad export cookie 14567566948027736287 [11326.687232] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11326.696328] LustreError: 317082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11327.544786] Lustre: server umount lustre-MDT0001 complete [11341.570722] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11345.059946] Key type lgssc unregistered [11345.393832] LNet: 320669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11345.401481] LNetError: 320669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11345.418087] LNet: Removed LNI 192.168.202.102@tcp [11346.470509] Key type .llcrypt unregistered [11346.472398] Key type ._llcrypt unregistered [11372.122480] Key type ._llcrypt registered [11372.124854] Key type .llcrypt registered [11372.262898] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11373.333910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11373.471429] alg: No test for adler32 (adler32-zlib) [11374.622867] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [11374.970449] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11376.695187] Key type lgssc registered [11377.784826] Lustre: Echo OBD driver; http://www.lustre.org/ [11388.774662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11388.809683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.536422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11395.754266] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11400.684393] LustreError: 322089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.701610] LustreError: 322089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11405.796297] LustreError: 322089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.424932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.837044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11412.552546] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11422.011866] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11429.739653] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11440.904473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11441.419898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11450.064436] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11459.368835] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11475.597296] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [11476.451828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.466035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.466261] Lustre: Skipped 1 previous similar message [11476.480429] Lustre: Skipped 1 previous similar message [11480.380605] Lustre: server umount lustre-MDT0000 complete [11480.661540] LustreError: 322084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.400406] LustreError: 322823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472493 with bad export cookie 5534460613924197454 [11485.421302] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11485.422507] LustreError: 322823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11485.789417] LustreError: 322084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.830525] LustreError: 322084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11485.847717] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [11486.691348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.718862] Lustre: Skipped 1 previous similar message [11490.720850] LustreError: 323337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11490.721892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11490.768206] LustreError: 323337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11490.773765] Lustre: Skipped 1 previous similar message [11492.320389] Lustre: server umount lustre-MDT0001 complete [11500.529387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11511.493337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.789025] LustreError: 321398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c45351ff x1874300641522176/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11531.379316] LustreError: 323922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.402444] LustreError: 323922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11531.475433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11531.552691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [11536.657980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11545.074412] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11547.171917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.702790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11552.764669] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11553.231737] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11561.608755] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11562.742984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11569.864264] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11572.136146] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11582.946783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11582.959650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.977407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11582.984292] Lustre: Skipped 1 previous similar message [11586.302533] Lustre: server umount lustre-OST0000 complete [11593.704559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11593.715196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11593.732350] Lustre: Skipped 1 previous similar message [11593.756548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11593.768373] Lustre: Skipped 3 previous similar messages [11597.234655] Lustre: server umount lustre-MDT0000 complete [11598.816225] LustreError: 325711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.839702] LustreError: 325711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11602.400966] LustreError: 322070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472610 with bad export cookie 5534460613924205098 [11602.407884] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.411764] LustreError: 322070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11602.906104] Lustre: server umount lustre-MDT0001 complete [11617.970934] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11621.255466] Key type lgssc unregistered [11621.575982] LNet: 328384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11621.585984] LNetError: 328384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11621.611784] LNet: Removed LNI 192.168.202.102@tcp [11622.447784] Key type .llcrypt unregistered [11622.454362] Key type ._llcrypt unregistered [11651.673883] Key type ._llcrypt registered [11651.675431] Key type .llcrypt registered [11651.804262] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11653.338873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11653.377718] alg: No test for adler32 (adler32-zlib) [11654.582371] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [11654.962949] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11656.655676] Key type lgssc registered [11658.156624] Lustre: Echo OBD driver; http://www.lustre.org/ [11666.929707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11666.954663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.512340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11673.307453] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11678.690290] LustreError: 329800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.719341] LustreError: 329800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11683.816674] LustreError: 329801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.875200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.270214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11689.478680] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11697.693455] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11704.255662] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11709.473499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.490177] Lustre: Skipped 1 previous similar message [11709.491419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11713.788223] Lustre: server umount lustre-MDT0000 complete [11714.534989] LustreError: 329797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.566642] LustreError: 329797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11718.021814] LustreError: 329780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472726 with bad export cookie 4690707370213317815 [11718.023468] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.031080] LustreError: 329780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11718.418124] Lustre: server umount lustre-MDT0001 complete [11736.201116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.190630] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11737.222595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11737.388106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.698443] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11757.229543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.321423] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11757.379535] Lustre: 332392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11757.687377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11765.419527] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11775.086627] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.116674] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11794.122073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.468265] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11794.491849] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11794.506386] LustreError: 334278:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [11794.524454] LustreError: 334278:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11794.534889] LustreError: 334278:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11794.663487] Lustre: server umount lustre-OST0000 complete [11794.666967] LustreError: 334278:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11807.210212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.513557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11807.793724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11814.176316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11816.488746] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11826.387268] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11838.236469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11838.447143] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11838.637157] Lustre: lustre-OST0001: new disk, initializing [11838.645480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11838.661373] Lustre: 335823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11838.839442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11845.280560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11845.295858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11845.494425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11848.589321] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11859.128731] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11866.080358] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11866.086879] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11866.095163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11866.117220] Lustre: Skipped 1 previous similar message [11866.118906] Lustre: Skipped 1 previous similar message [11871.216079] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11871.237279] Lustre: Skipped 2 previous similar messages [11876.348961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11876.371199] Lustre: Skipped 1 previous similar message [11877.343868] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11877.679685] Lustre: server umount lustre-OST0001 complete [11881.463538] LustreError: 334747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.488601] LustreError: 334747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11886.051519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11886.053017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11886.067367] LustreError: Skipped 2 previous similar messages [11886.073111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11886.093962] Lustre: Skipped 1 previous similar message [11886.119087] Lustre: Skipped 1 previous similar message [11886.560142] LustreError: 334744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11886.592952] LustreError: 334744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11896.803611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11896.808089] LustreError: 334744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11896.808607] Lustre: Skipped 3 previous similar messages [11896.840167] LustreError: 334744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11897.311360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11897.595947] Lustre: server umount lustre-OST0000 complete [11906.033417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11906.058064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11909.046707] Lustre: server umount lustre-MDT0000 complete [11913.696933] LustreError: 335822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472921 with bad export cookie 4690707370213318368 [11913.706648] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11913.716568] LustreError: 335822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11914.399586] Lustre: server umount lustre-MDT0001 complete [11928.682324] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11932.227622] Key type lgssc unregistered [11932.651990] LNet: 337788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11932.654585] LNetError: 337788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11932.665207] LNet: Removed LNI 192.168.202.102@tcp [11933.873364] Key type .llcrypt unregistered [11933.875908] Key type ._llcrypt unregistered [12015.594756] Key type ._llcrypt registered [12015.599862] Key type .llcrypt registered [12015.714734] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12032.921779] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12034.627307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12034.667733] alg: No test for adler32 (adler32-zlib) [12035.904707] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12036.214366] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12037.935208] Key type lgssc registered [12039.201345] Lustre: Echo OBD driver; http://www.lustre.org/ [12080.947232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12081.005948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.471288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12082.508593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12082.667442] Lustre: lustre-MDT0000: new disk, initializing [12082.793758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12082.831157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12088.621385] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12101.381045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.509279] Lustre: 342727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12101.567782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12101.571258] Lustre: Skipped 1 previous similar message [12101.671320] Lustre: lustre-MDT0001: new disk, initializing [12101.847336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12101.925050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12101.948327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12106.256422] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12121.475876] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12129.533695] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12137.802794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.193571] Lustre: lustre-OST0000: new disk, initializing [12138.200748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12138.207195] Lustre: 344694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12138.312831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12145.265079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12145.279173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12145.404641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12145.787838] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12157.285536] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12162.927385] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12163.280404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12168.622370] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12168.837040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12171.234590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12171.239569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.254498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12171.292793] Lustre: Skipped 1 previous similar message [12176.353261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12176.369910] Lustre: Skipped 2 previous similar messages [12177.266958] Lustre: server umount lustre-OST0000 complete [12181.475179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12181.498103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12181.502921] Lustre: Skipped 1 previous similar message [12186.594211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12186.601385] Lustre: Skipped 1 previous similar message [12187.663785] Lustre: server umount lustre-MDT0000 complete [12192.160983] LustreError: 342718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473200 with bad export cookie 3648616911629363501 [12192.168731] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12192.181490] LustreError: 342718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12192.914771] Lustre: server umount lustre-MDT0001 complete [12203.730238] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:20:11 (1787473211) [12214.158206] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:20:21 (1787473221) [12226.656498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.370970] LustreError: 347452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12227.401353] LustreError: 347452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12227.503591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12232.672824] LustreError: 347451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12234.135630] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12237.802958] LustreError: 347452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12242.919568] LustreError: 347451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.059118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12247.597467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12253.975696] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12263.971932] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12271.821684] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12280.429678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.049788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12289.208644] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12300.495350] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12308.691939] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [12311.037782] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [12312.858428] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [12314.603609] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [12317.069793] Lustre: Failing over lustre-MDT0000 [12317.313658] LustreError: 350226:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000e9c96d4f) refcount nonzero (2) after lock cleanup; forcing cleanup. [12317.413353] Lustre: server umount lustre-MDT0000 complete [12317.816913] LustreError: 347446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.200234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12319.204772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12319.214711] LustreError: Skipped 1 previous similar message [12319.237848] Lustre: Skipped 2 previous similar messages [12328.034888] LustreError: 347447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.064962] LustreError: 347447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12335.583292] Lustre: 340947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787473327/real 1787473327] req@0000000087ba637f x1874301334662656/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787473343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12335.611911] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12337.660300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12344.403372] LustreError: 347448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.441562] LustreError: 347448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [12345.834502] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x32a27dbffb26b0b9 [12345.886810] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [12346.197862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12346.315401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12349.542608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12351.491571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12351.535406] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [12351.608236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12352.177358] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12363.457384] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12365.517653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12370.279618] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [12372.715401] Lustre: Failing over lustre-MDT0000 [12373.010737] Lustre: server umount lustre-MDT0000 complete [12377.056248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12377.061116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.067421] LustreError: 350343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12377.078073] Lustre: Skipped 3 previous similar messages [12377.115890] LustreError: 350343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12393.444944] Lustre: 340947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787473385/real 1787473385] req@00000000d6da26e8 x1874301334686720/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787473401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12393.480672] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12394.604912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12403.683516] LustreError: 340946:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008358d14b x1874301334692736/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12403.938249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12404.021670] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12405.848746] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12409.322793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12409.333911] Lustre: Skipped 2 previous similar messages [12409.400828] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [12409.453265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12409.925234] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12423.064393] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12425.512493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12434.915100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.916979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12434.931124] Lustre: Skipped 2 previous similar messages [12434.949946] Lustre: Skipped 2 previous similar messages [12438.282757] Lustre: server umount lustre-MDT0000 complete [12442.916624] LustreError: 347432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473450 with bad export cookie 3648616911629366224 [12442.921888] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12442.926371] LustreError: 347432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12443.579294] Lustre: server umount lustre-MDT0001 complete [12459.107718] Lustre: server umount lustre-OST0000 complete [12478.531341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.979461] LustreError: 353801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12478.996280] LustreError: 353801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [12479.172528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12484.759058] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12495.771483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.948745] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12510.965361] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12518.933982] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12528.086255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12536.015297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12546.233249] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12549.000388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12559.331251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12559.342328] Lustre: Skipped 1 previous similar message [12559.346816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12559.350611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12563.790786] Lustre: server umount lustre-OST0000 complete [12574.648364] Lustre: server umount lustre-MDT0000 complete [12579.044826] LustreError: 353783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473587 with bad export cookie 3648616911629366966 [12579.061400] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12579.495509] Lustre: server umount lustre-MDT0001 complete [12591.803648] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12595.278143] Key type lgssc unregistered [12595.679795] LNet: 357338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12595.687259] LNetError: 357338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12596.775479] LNet: Removed LNI 192.168.202.102@tcp [12597.791515] Key type .llcrypt unregistered [12597.797053] Key type ._llcrypt unregistered [12630.390515] Key type ._llcrypt registered [12630.393123] Key type .llcrypt registered [12630.538332] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12631.853138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12631.879659] alg: No test for adler32 (adler32-zlib) [12633.032709] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12633.308668] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12635.007167] Key type lgssc registered [12636.961295] Lustre: Echo OBD driver; http://www.lustre.org/ [12646.666690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12646.703167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.252710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12653.217296] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12658.676495] LustreError: 358858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.698370] LustreError: 358858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12663.779275] LustreError: 358859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.923781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12664.269877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12671.468922] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12680.873304] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12686.773678] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12693.794434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12694.157134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12700.996920] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12708.646247] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12718.785740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12729.313055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12729.322420] Lustre: Skipped 1 previous similar message [12729.346651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.351505] Lustre: Skipped 1 previous similar message [12734.437217] LustreError: 358859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.566736] Lustre: server umount lustre-MDT0000 complete [12738.811471] LustreError: 358839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473746 with bad export cookie 13158271205996118814 [12738.818540] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.236319] Lustre: server umount lustre-MDT0001 complete [12755.067632] Lustre: server umount lustre-OST0000 complete [12771.307421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12771.693059] LustreError: 362597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12771.715287] LustreError: 362597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12771.793988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12776.943983] LustreError: 362598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12777.402971] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12786.149805] LustreError: 362598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.171338] LustreError: 362598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12787.562736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12787.885789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12793.553338] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12802.897438] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12810.294779] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12818.064636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12818.422899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12825.531230] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12835.680150] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12841.983883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12862.433908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12862.440609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.451889] Lustre: Skipped 3 previous similar messages [12862.459700] Lustre: Skipped 1 previous similar message [12867.074069] Lustre: server umount lustre-MDT0000 complete [12867.553419] LustreError: 362598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.579530] LustreError: 362598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12871.481336] LustreError: 362579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473879 with bad export cookie 13158271205996119927 [12871.483293] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12871.506410] LustreError: 362579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12872.063584] Lustre: server umount lustre-MDT0001 complete [12887.754895] Lustre: server umount lustre-OST0000 complete [12906.398493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.927038] LustreError: 366533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12907.015094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12912.655671] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12923.219025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12930.091344] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12941.071433] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12948.638985] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12955.863418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12956.225948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12956.231924] Lustre: Skipped 1 previous similar message [12963.145258] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12972.827933] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12977.846854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12983.663915] Lustre: server umount lustre-OST0000 complete [12988.385094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12988.390616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12988.411600] Lustre: Skipped 1 previous similar message [12988.441821] Lustre: Skipped 2 previous similar messages [12993.518872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12993.526069] Lustre: Skipped 1 previous similar message [12998.626663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.636643] Lustre: Skipped 1 previous similar message [12998.694191] Lustre: server umount lustre-MDT0000 complete [13000.671226] Lustre: 358169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787473992/real 1787473992] req@0000000046a774fd x1874301960686720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787474008 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13000.699683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.731594] LustreError: 366534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.746650] LustreError: 366534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13003.040763] LustreError: 366515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474011 with bad export cookie 13158271205996120949 [13003.053219] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13003.509749] Lustre: server umount lustre-MDT0001 complete [13015.838819] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13019.183363] Key type lgssc unregistered [13019.566226] LNet: 370069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13019.578300] LNetError: 370069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13019.607599] LNet: Removed LNI 192.168.202.102@tcp [13020.770526] Key type .llcrypt unregistered [13020.772179] Key type ._llcrypt unregistered [13049.889931] Key type ._llcrypt registered [13049.891371] Key type .llcrypt registered [13050.034269] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13052.179709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13052.234209] alg: No test for adler32 (adler32-zlib) [13053.580103] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [13053.805494] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13055.447187] Key type lgssc registered [13057.010143] Lustre: Echo OBD driver; http://www.lustre.org/ [13065.980204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13065.997844] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13066.003970] LustreError: 371513:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [13066.023788] LustreError: 371513:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [13066.033371] LustreError: 371513:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [13066.047237] LustreError: 371513:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [13066.061867] LustreError: 371513:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [13074.572845] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [13074.584694] LustreError: 371910:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [13074.625681] LustreError: 371910:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [13074.638497] LustreError: 371910:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [13074.657477] LustreError: 371910:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [13074.671612] LustreError: 371910:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [13099.085994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13102.292690] Key type lgssc unregistered [13102.686777] LNet: 372917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13102.695340] LNetError: 372917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13102.723221] LNet: Removed LNI 192.168.202.102@tcp [13103.606316] Key type .llcrypt unregistered [13103.608039] Key type ._llcrypt unregistered [13125.175035] Key type ._llcrypt registered [13125.175954] Key type .llcrypt registered [13125.264729] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13144.677974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13146.934517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13146.995864] alg: No test for adler32 (adler32-zlib) [13148.418761] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [13149.003862] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13150.791288] Key type lgssc registered [13152.878762] Lustre: Echo OBD driver; http://www.lustre.org/ [13192.161964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13192.192423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13193.447129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13193.467873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13193.536730] Lustre: lustre-MDT0000: new disk, initializing [13193.639937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13193.653793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13198.869949] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13211.215577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13211.383545] Lustre: 376077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13211.453872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13211.459468] Lustre: Skipped 1 previous similar message [13211.562578] Lustre: lustre-MDT0001: new disk, initializing [13211.635484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13211.707876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13211.731372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13216.934569] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13228.191804] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13234.626631] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13242.136302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13242.501981] Lustre: lustre-OST0000: new disk, initializing [13242.506154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13242.513682] Lustre: 378046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13242.616332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13249.337377] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13251.657341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13251.664390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13251.774474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13260.655427] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13266.615158] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13266.925428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13272.146740] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13272.485096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13277.676259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13277.685585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13277.693924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13277.738074] Lustre: Skipped 1 previous similar message [13280.937295] Lustre: server umount lustre-OST0000 complete [13287.911319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13287.925033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13287.929049] Lustre: Skipped 1 previous similar message [13287.942046] Lustre: Skipped 2 previous similar messages [13291.425441] Lustre: server umount lustre-MDT0000 complete [13294.783062] LustreError: 376068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474302 with bad export cookie 18302733007036722144 [13294.786708] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13294.796064] LustreError: 376068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13295.383451] Lustre: server umount lustre-MDT0001 complete [13306.846811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [13309.275204] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:38:36 (1787474316) [13319.661306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13320.779612] LustreError: 380508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13320.817251] LustreError: 380508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13320.889523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13325.564778] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13326.311997] LustreError: 380509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13330.413399] LustreError: 380508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.522652] LustreError: 380509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.645067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13336.307241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13341.827866] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13350.148598] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13358.461829] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13367.270662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13367.751097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13375.677580] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13386.238316] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13396.725621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.011360] Lustre: lustre-OST0001: new disk, initializing [13397.024759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13397.033914] Lustre: 383347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13397.208844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13404.228519] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13404.847267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13404.860553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13404.981546] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13415.617704] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13423.752851] Lustre: lustre-OST0001: Not available for connect from 192.168.202.2@tcp (stopping) [13425.637808] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13425.673330] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13425.680295] LustreError: Skipped 2 previous similar messages [13426.904424] Lustre: server umount lustre-OST0001 complete [13428.820423] LustreError: 382330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13441.011716] LustreError: 382333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13441.030920] LustreError: 382333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13446.115426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13446.138608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13446.142540] Lustre: Skipped 2 previous similar messages [13446.157615] Lustre: Skipped 3 previous similar messages [13451.244655] Lustre: server umount lustre-OST0000 complete [13456.352415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13456.382308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13456.396149] Lustre: Skipped 1 previous similar message [13462.644816] Lustre: server umount lustre-MDT0000 complete [13466.593398] LustreError: 380505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13466.623034] LustreError: 380505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13466.804216] LustreError: 380489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474474 with bad export cookie 18302733007036723026 [13466.814486] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13473.393080] Lustre: server umount lustre-MDT0001 complete [13486.732927] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13490.453735] Key type lgssc unregistered [13490.763823] LNet: 385336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13490.772948] LNetError: 385336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13491.821667] LNet: Removed LNI 192.168.202.102@tcp [13492.819758] Key type .llcrypt unregistered [13492.823227] Key type ._llcrypt unregistered [13550.022804] Key type ._llcrypt registered [13550.025274] Key type .llcrypt registered [13550.136126] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13551.206599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13551.422314] alg: No test for adler32 (adler32-zlib) [13552.597639] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [13552.926121] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13554.647369] Key type lgssc registered [13555.936949] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.015197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13565.027361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.362605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13566.386629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13566.566588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13571.359582] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13581.762694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13581.930065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13581.967090] Lustre: 388137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13582.268846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13587.796442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13597.708450] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13606.028828] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13611.493655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13611.506820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13611.511856] Lustre: Skipped 1 previous similar message [13616.017514] Lustre: server umount lustre-MDT0000 complete [13620.373028] LustreError: 388131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474628 with bad export cookie 18069619617006035557 [13620.379297] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13620.390474] LustreError: 388131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13620.878664] Lustre: server umount lustre-MDT0001 complete [13632.814230] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:44:00 (1787474640) [13641.512136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.902270] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13641.915943] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13641.983270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13646.696852] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13647.331645] LustreError: 390654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13652.449161] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13657.492423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13657.569368] LustreError: 390654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13658.145806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13663.581122] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13673.353643] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13681.151531] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13690.534418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13690.804982] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13690.961178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13699.242991] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13708.353284] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13718.655905] Lustre: Permanently deactivating lustre-OST0000 [13718.701864] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [13727.818965] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13729.328276] LustreError: 393642:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [13729.341451] LustreError: 393642:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [13729.349777] LustreError: 393642:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13732.981477] LustreError: 393691:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [13732.986859] LustreError: 393691:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13741.587957] Lustre: server umount lustre-OST0000 complete [13745.426057] LustreError: 393941:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [13745.430173] LustreError: 393941:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13749.236057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.246626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.269521] Lustre: Skipped 1 previous similar message [13749.286630] Lustre: Skipped 2 previous similar messages [13754.339581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13754.361531] Lustre: Skipped 1 previous similar message [13757.504481] Lustre: server umount lustre-MDT0000 complete [13759.460371] LustreError: 390654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13759.508239] LustreError: 390654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13762.352464] LustreError: 390633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474770 with bad export cookie 18069619617006036145 [13762.363275] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13762.366203] LustreError: 390633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13772.937231] Lustre: server umount lustre-MDT0001 complete [13784.920346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13791.045692] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13794.200683] LustreError: 395102:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [13801.658969] Lustre: 395442:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.102@tcp [13803.887438] Lustre: server umount MGS complete [13818.215996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.826980] LustreError: 396166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13818.853539] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13818.865101] Lustre: Skipped 1 previous similar message [13818.944863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13824.784900] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13834.580705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13834.861342] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13835.022740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13840.304432] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13848.657734] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13855.777756] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13857.951345] Lustre: Permanently reactivating lustre-OST0000 [13857.981509] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13857.992474] Lustre: Skipped 2 previous similar messages [13883.871351] LustreError: 397897:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13893.426232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13894.002475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13901.384795] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13910.418947] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13918.113746] Lustre: server umount lustre-OST0000 complete [13922.279939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13922.288154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13922.312189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13924.836403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13929.953392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13929.964252] Lustre: Skipped 3 previous similar messages [13932.225738] Lustre: server umount lustre-MDT0000 complete [13934.047219] Lustre: 387460:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787474926/real 1787474926] req@00000000a9fad577 x1874302924330880/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787474942 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13934.096628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13934.107680] LustreError: 396166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13934.126769] LustreError: 396166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13935.071100] Lustre: 387461:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787474927/real 1787474927] req@0000000021586ed3 x1874302924331648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787474943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13937.463085] LustreError: 396146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474945 with bad export cookie 18069619617006037195 [13937.464594] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13937.482460] LustreError: 396146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13938.020459] Lustre: server umount lustre-MDT0001 complete [13952.067480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13954.905367] Key type lgssc unregistered [13955.223760] LNet: 400537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13955.232518] LNetError: 400537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13955.261133] LNet: Removed LNI 192.168.202.102@tcp [13956.419329] Key type .llcrypt unregistered [13956.422870] Key type ._llcrypt unregistered [13978.754432] Key type ._llcrypt registered [13978.756793] Key type .llcrypt registered [13978.976840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13999.953802] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14001.645179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14002.091311] alg: No test for adler32 (adler32-zlib) [14003.181962] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [14003.425893] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14005.119324] Key type lgssc registered [14005.789780] Lustre: Echo OBD driver; http://www.lustre.org/ [14048.088865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14048.136335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14049.624247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14049.655846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14049.753972] Lustre: lustre-MDT0000: new disk, initializing [14049.883800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14049.918583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14055.434331] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14069.244502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14069.384537] Lustre: 403697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14069.459776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14069.463453] Lustre: Skipped 1 previous similar message [14069.541315] Lustre: lustre-MDT0001: new disk, initializing [14069.698469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14069.751277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14069.773647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14074.946498] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14087.303608] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14096.305606] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14107.031100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14107.443377] Lustre: lustre-OST0000: new disk, initializing [14107.447852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14107.454438] Lustre: 405667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14107.541462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14115.521484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14115.546793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14115.767579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14115.992176] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14126.657985] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14132.429772] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14132.818764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14138.811613] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14139.093196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14146.537879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14146.539854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14146.578899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14146.599555] Lustre: Skipped 1 previous similar message [14148.226895] Lustre: server umount lustre-OST0000 complete [14156.771112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14156.771973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14156.780448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14156.793734] Lustre: Skipped 2 previous similar messages [14159.189663] Lustre: server umount lustre-MDT0000 complete [14164.615907] LustreError: 403689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787475172 with bad export cookie 4572944329605770545 [14164.633088] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14165.387629] Lustre: server umount lustre-MDT0001 complete [14180.798805] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [14184.366896] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:53:10 (1787475190) [14194.510685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14195.199981] LustreError: 408134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14195.229126] LustreError: 408134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14195.399273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14200.801227] LustreError: 408135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14202.159414] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14205.929142] LustreError: 408134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14211.044829] LustreError: 408135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14215.368062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14216.086122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14221.757591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14232.846630] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14241.289682] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14248.917458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14249.377952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14256.836365] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14265.771244] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14268.139105] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [14270.941540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [14277.088541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14277.091000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14277.134031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14277.140342] Lustre: Skipped 1 previous similar message [14281.189074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14281.190553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14281.203564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14281.226919] Lustre: Skipped 1 previous similar message [14283.059296] Lustre: server umount lustre-OST0000 complete [14287.851991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14287.870918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14287.898815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14296.544989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14296.553971] Lustre: Skipped 3 previous similar messages [14302.175196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14302.537513] Lustre: server umount lustre-MDT0000 complete [14306.790310] LustreError: 408875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14306.823454] LustreError: 408875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14307.750094] LustreError: 408115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787475315 with bad export cookie 4572944329605771427 [14307.759640] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14307.775166] LustreError: 408115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14308.499931] Lustre: server umount lustre-MDT0001 complete [14325.453247] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14329.359226] Key type lgssc unregistered [14329.733430] LNet: 411869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14329.742461] LNetError: 411869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14329.762890] LNet: Removed LNI 192.168.202.102@tcp [14331.019643] Key type .llcrypt unregistered [14331.021356] Key type ._llcrypt unregistered [14361.058796] Key type ._llcrypt registered [14361.061168] Key type .llcrypt registered [14361.212423] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14361.499624] Key type .llcrypt unregistered [14361.503887] Key type ._llcrypt unregistered [14381.021901] Key type ._llcrypt registered [14381.032423] Key type .llcrypt registered [14381.144833] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14382.934354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14382.983850] alg: No test for adler32 (adler32-zlib) [14384.049328] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [14384.286285] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14385.943310] Key type lgssc registered [14387.551915] Lustre: Echo OBD driver; http://www.lustre.org/ [14395.764515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14395.791768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14397.133443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14401.671598] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14407.650600] LustreError: 414091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14407.664293] LustreError: 414091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14411.275487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14411.491334] LustreError: 414090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14411.517960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14416.890566] LustreError: 414084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14416.919672] LustreError: 414084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14420.742884] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14421.985571] LustreError: 414086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14422.011327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [14422.013622] LustreError: 414086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14427.117065] LustreError: 414091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14427.153193] LustreError: 414091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14430.851575] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14437.349980] LustreError: 414087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14437.373090] LustreError: 414087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14440.283447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14440.987575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14448.402484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14461.418885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14461.427768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14461.439219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14465.675452] Lustre: server umount lustre-OST0000 complete [14471.649166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14471.650499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14471.666317] Lustre: Skipped 1 previous similar message [14471.668267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14471.668273] Lustre: Skipped 1 previous similar message [14472.868593] Lustre: server umount lustre-MDT0000 complete [14476.770849] LustreError: 414087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14476.792136] LustreError: 414087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14478.846360] LustreError: 414070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787475486 with bad export cookie 14819794029657478949 [14478.855497] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14479.584829] Lustre: server umount lustre-MDT0001 complete [14494.372642] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14498.498206] Key type lgssc unregistered [14498.849349] LNet: 417242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14498.863327] LNetError: 417242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14498.882700] LNet: Removed LNI 192.168.202.102@tcp [14499.896057] Key type .llcrypt unregistered [14499.897264] Key type ._llcrypt unregistered [14530.574361] Key type ._llcrypt registered [14530.577453] Key type .llcrypt registered [14530.770432] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14532.353500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14532.703970] alg: No test for adler32 (adler32-zlib) [14533.779770] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [14534.075670] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14535.911264] Key type lgssc registered [14537.350876] Lustre: Echo OBD driver; http://www.lustre.org/ [14546.037826] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14546.063830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14561.631526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [14566.751498] LustreError: 418634:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [14593.375247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14600.047711] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14608.699054] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14616.415298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14617.047660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14617.160803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [14622.177937] LustreError: 418659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14622.192476] LustreError: 418659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14622.533493] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14627.299409] LustreError: 418660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14628.324449] LustreError: 419672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14628.355917] LustreError: 419672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14632.632677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14633.142084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14638.393205] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14646.171550] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14652.494203] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14663.136247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14663.148623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14663.170180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14663.648142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14663.663139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14663.685746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14667.412540] Lustre: server umount lustre-OST0000 complete [14673.851531] Lustre: server umount lustre-MDT0000 complete [14673.902238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14673.918170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14673.936819] LustreError: 419675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14677.942062] LustreError: 418642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787475686 with bad export cookie 18176676892955510384 [14677.947338] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14677.956159] LustreError: 418642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14678.375753] Lustre: server umount lustre-MDT0001 complete [14692.940911] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14698.290638] Key type lgssc unregistered [14698.559326] LNet: 422218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14698.562793] LNetError: 422218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14698.570749] LNet: Removed LNI 192.168.202.102@tcp [14699.806145] Key type .llcrypt unregistered [14699.819656] Key type ._llcrypt unregistered [14732.580718] Key type ._llcrypt registered [14732.583655] Key type .llcrypt registered [14732.693214] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14733.701766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14733.955291] alg: No test for adler32 (adler32-zlib) [14735.115730] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [14735.364338] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14737.047176] Key type lgssc registered [14738.441082] Lustre: Echo OBD driver; http://www.lustre.org/ [14746.489560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14746.509578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14748.161526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14753.705628] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14758.391761] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14758.414015] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14762.362928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14762.649396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14766.949955] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14776.041572] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14782.300314] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14789.329417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14789.825782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14793.930576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [14796.462903] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14804.118735] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14808.544443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14808.551494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14808.572267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14810.435739] Lustre: server umount lustre-MDT0000 complete [14811.280629] LustreError: 423634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.642025] LustreError: 424653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.646784] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14812.650214] LustreError: 424653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14812.657090] Lustre: Skipped 1 previous similar message [14812.860953] LustreError: 423616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.2@tcp arrived at 1787475820 with bad export cookie 6009651478015732426 [14815.059278] LustreError: 423617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787475823 with bad export cookie 6009651478015731530 [14815.064290] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14815.073101] LustreError: 423617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14817.771547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14817.785212] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14817.796290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14817.821213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14817.822137] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14820.774137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14821.329812] Lustre: server umount lustre-OST0000 complete [14825.905453] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14825.929634] LustreError: 423637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14829.694382] Lustre: server umount lustre-MDT0001 complete [14842.563150] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14845.652902] Key type lgssc unregistered [14846.137891] LNet: 427324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14846.146539] LNetError: 427324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14846.165223] LNet: Removed LNI 192.168.202.102@tcp [14847.426968] Key type .llcrypt unregistered [14847.436157] Key type ._llcrypt unregistered [14876.524711] Key type ._llcrypt registered [14876.529933] Key type .llcrypt registered [14876.756476] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14878.351272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14878.452419] alg: No test for adler32 (adler32-zlib) [14880.233232] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [14881.080980] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14882.831885] Key type lgssc registered [14885.388272] Lustre: Echo OBD driver; http://www.lustre.org/ [14895.220560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14895.258132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14896.778475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14901.229527] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14906.340559] LustreError: 428741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14906.352685] LustreError: 428741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14910.541198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14910.960892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14915.965594] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14924.145123] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14930.525512] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14937.559505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14938.002285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14941.068398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14944.690114] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14952.518118] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14960.623988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14960.638874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14964.313034] Lustre: server umount lustre-MDT0001 complete [14965.731159] LustreError: 428736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14965.771835] LustreError: 428736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14970.850744] LustreError: 428742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14970.907880] LustreError: 428742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14975.970192] LustreError: 428741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14975.971957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14976.004703] Lustre: Skipped 1 previous similar message [14976.015031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14976.021102] Lustre: Skipped 2 previous similar messages [14978.129659] Lustre: server umount lustre-OST0000 complete [14981.089256] LustreError: 428742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14983.838637] Lustre: server umount lustre-MDT0000 complete [15001.543294] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15004.829269] Key type lgssc unregistered [15005.327774] LNet: 432429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15005.334238] LNetError: 432429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15005.359521] LNet: Removed LNI 192.168.202.102@tcp [15006.277901] Key type .llcrypt unregistered [15006.280850] Key type ._llcrypt unregistered [15033.252967] Key type ._llcrypt registered [15033.260856] Key type .llcrypt registered [15033.385867] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [15049.746230] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15050.956188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15050.975173] alg: No test for adler32 (adler32-zlib) [15052.207843] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [15052.381783] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [15053.983160] Key type lgssc registered [15055.154886] Lustre: Echo OBD driver; http://www.lustre.org/ [15111.653494] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15130.598633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15130.632664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15132.062318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15132.097772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15132.203978] Lustre: lustre-MDT0000: new disk, initializing [15132.329625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15132.360321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15137.139330] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15154.213513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15154.485396] Lustre: lustre-OST0000: new disk, initializing [15154.491442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15154.499803] Lustre: Skipped 1 previous similar message [15154.510686] Lustre: 437491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15154.575595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15162.109476] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15163.048583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15163.059665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15163.093933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15173.403899] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15191.286740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15191.492733] Lustre: 436475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15191.571352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15191.695282] Lustre: lustre-MDT0001: new disk, initializing [15191.896977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15191.968695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [15191.987947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [15197.121956] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15207.772341] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [15208.022915] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [15213.162795] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [15213.401823] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [15220.705195] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [15220.712039] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15220.717285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15220.732232] Lustre: Skipped 1 previous similar message [15222.366058] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [15222.370414] Lustre: Skipped 1 previous similar message [15225.420313] Lustre: server umount lustre-MDT0001 complete [15230.978593] LustreError: 436482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15231.020429] LustreError: 436482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [15236.071374] LustreError: 438830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15236.089318] LustreError: 438830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15241.210462] LustreError: 436486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15241.253944] LustreError: 436486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15246.312838] LustreError: 438830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15246.315207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15246.362698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15251.426171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15251.428197] LustreError: 436486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15256.560941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15260.127161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15260.298574] Lustre: server umount lustre-OST0000 complete [15261.676026] LustreError: 436486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15261.691603] LustreError: 436486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15270.810164] Lustre: server umount lustre-MDT0000 complete [15283.642516] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15285.962306] Key type lgssc unregistered [15286.202517] LNet: 441245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15286.207903] LNetError: 441245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15286.230881] LNet: Removed LNI 192.168.202.102@tcp [15287.307177] Key type .llcrypt unregistered [15287.309687] Key type ._llcrypt unregistered [15336.489907] Key type ._llcrypt registered [15336.492899] Key type .llcrypt registered [15336.678886] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15337.005545] Key type .llcrypt unregistered [15337.007156] Key type ._llcrypt unregistered [15360.579540] Key type ._llcrypt registered [15360.581771] Key type .llcrypt registered [15360.728574] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15361.988479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15362.020598] alg: No test for adler32 (adler32-zlib) [15363.222478] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [15363.519490] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [15365.287196] Key type lgssc registered [15367.233703] Lustre: Echo OBD driver; http://www.lustre.org/ [15403.122555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15403.150118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15404.528322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15404.592402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15404.737463] Lustre: lustre-MDT0000: new disk, initializing [15404.839205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15404.856889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15410.408413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15421.631564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15421.801876] Lustre: 444630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15421.863334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15421.867247] Lustre: Skipped 1 previous similar message [15421.887140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15421.975827] Lustre: lustre-MDT0001: new disk, initializing [15422.119940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15422.183434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15422.208332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15427.544772] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15438.019169] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15445.899810] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15454.006711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15454.366966] Lustre: lustre-OST0000: new disk, initializing [15454.375051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15454.383393] Lustre: 446602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15454.497612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15460.966943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15460.973728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15461.144293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15461.645426] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15472.508273] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15481.830585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15481.841501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15481.848358] Lustre: Skipped 1 previous similar message [15481.858810] Lustre: Skipped 2 previous similar messages [15487.119656] Lustre: server umount lustre-MDT0000 complete [15491.246497] LustreError: 444622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476499 with bad export cookie 16334299626621234240 [15491.249725] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15491.259535] LustreError: 444622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15491.881992] Lustre: server umount lustre-MDT0001 complete [15506.497528] Lustre: server umount lustre-OST0000 complete [15517.593400] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:15:25 (1787476525) [15526.033641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15526.570991] LustreError: 448628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15526.594824] LustreError: 448628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15526.660898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15532.008973] LustreError: 448629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15532.195058] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15536.161610] LustreError: 448628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15541.219447] LustreError: 448629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15543.450733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15543.932473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15548.418709] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15554.767717] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15561.613400] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15580.665933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15580.949403] Lustre: Found index 0 for lustre-OST0000, updating log [15581.220073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15589.487155] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15597.646520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [15598.094282] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15604.705749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15604.712199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15604.728302] Lustre: Skipped 1 previous similar message [15604.747494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15609.976516] Lustre: server umount lustre-OST0000 complete [15627.241080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15627.259748] Lustre: Skipped 1 previous similar message [15627.263869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15627.270343] Lustre: Skipped 4 previous similar messages [15631.394176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15631.407767] Lustre: Skipped 1 previous similar message [15632.417948] LustreError: 448628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15632.443687] LustreError: 448628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15632.476608] Lustre: server umount lustre-MDT0000 complete [15637.022881] LustreError: 450480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476645 with bad export cookie 16334299626621235633 [15637.029053] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15637.035286] LustreError: 450480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15637.303296] Lustre: server umount lustre-MDT0001 complete [15640.032314] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [15645.628866] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:17:33 (1787476653) [16275.683574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16276.367493] LustreError: 455038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16276.529017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16281.574930] LustreError: 455039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16283.008529] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16286.691801] LustreError: 455038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16292.836722] LustreError: 455039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16292.863835] LustreError: 455039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16294.467876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16294.827033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16299.128686] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16307.454692] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16315.092858] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16330.280972] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [16337.527909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16337.940383] Lustre: Found index 0 for lustre-OST0000, updating log [16338.317676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16345.869702] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16354.419627] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16357.857805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16357.866584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16357.885398] Lustre: Skipped 1 previous similar message [16357.897389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16363.336723] Lustre: server umount lustre-OST0000 complete [16368.743533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16368.750738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16368.765034] Lustre: Skipped 1 previous similar message [16368.784335] Lustre: Skipped 1 previous similar message [16370.848344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16370.859666] Lustre: Skipped 3 previous similar messages [16374.110303] Lustre: server umount lustre-MDT0000 complete [16374.967591] LustreError: 457640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16375.002866] LustreError: 457640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16378.265785] LustreError: 455020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477386 with bad export cookie 16334299626621236718 [16378.278480] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16378.283544] LustreError: 455020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16378.651319] Lustre: server umount lustre-MDT0001 complete [16390.687380] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [16411.317416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [16421.266536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16459.609284] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:31:07 (1787477467) [16477.657163] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16480.940334] Key type lgssc unregistered [16481.215807] LNet: 461657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16481.220717] LNetError: 461657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16599.017442] LNet: Removed LNI 192.168.202.102@tcp [16599.828204] Key type .llcrypt unregistered [16599.830765] Key type ._llcrypt unregistered [16617.922618] Key type ._llcrypt registered [16617.930505] Key type .llcrypt registered [16618.131552] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16619.448432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16619.692441] alg: No test for adler32 (adler32-zlib) [16620.886835] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [16621.324978] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16623.039167] Key type lgssc registered [16625.207760] Lustre: Echo OBD driver; http://www.lustre.org/ [16630.821031] LNet: Added LNI 192.168.202.102@tcp100 [8/256/0/180] [16632.099738] LNet: Added LNI 192.168.202.102@tcp101 [8/256/0/180] [16633.700067] LNet: Added LNI 192.168.202.102@tcp102 [8/256/0/180] [16634.986116] LNet: Added LNI 192.168.202.102@tcp103 [8/256/0/180] [16636.246678] LNet: Added LNI 192.168.202.102@tcp104 [8/256/0/180] [16637.784064] LNet: Added LNI 192.168.202.102@tcp105 [8/256/0/180] [16638.831992] LNet: Added LNI 192.168.202.102@tcp106 [8/256/0/180] [16640.340683] LNet: Added LNI 192.168.202.102@tcp107 [8/256/0/180] [16641.739736] LNet: Added LNI 192.168.202.102@tcp108 [8/256/0/180] [16643.014605] LNet: Added LNI 192.168.202.102@tcp109 [8/256/0/180] [16644.440704] LNet: Added LNI 192.168.202.102@tcp110 [8/256/0/180] [16645.543511] LNet: Added LNI 192.168.202.102@tcp111 [8/256/0/180] [16646.188157] LNet: Added LNI 192.168.202.102@tcp112 [8/256/0/180] [16647.004882] LNet: Added LNI 192.168.202.102@tcp113 [8/256/0/180] [16647.826844] LNet: Added LNI 192.168.202.102@tcp114 [8/256/0/180] [16648.597769] LNet: Added LNI 192.168.202.102@tcp115 [8/256/0/180] [16649.722412] LNet: Added LNI 192.168.202.102@tcp116 [8/256/0/180] [16650.720865] LNet: Added LNI 192.168.202.102@tcp117 [8/256/0/180] [16651.548610] LNet: Added LNI 192.168.202.102@tcp118 [8/256/0/180] [16652.320779] LNet: Added LNI 192.168.202.102@tcp119 [8/256/0/180] [16653.162977] LNet: Added LNI 192.168.202.102@tcp120 [8/256/0/180] [16653.933825] LNet: Added LNI 192.168.202.102@tcp121 [8/256/0/180] [16654.842051] LNet: Added LNI 192.168.202.102@tcp122 [8/256/0/180] [16655.801980] LNet: Added LNI 192.168.202.102@tcp123 [8/256/0/180] [16656.844894] LNet: Added LNI 192.168.202.102@tcp124 [8/256/0/180] [16657.600120] LNet: Added LNI 192.168.202.102@tcp125 [8/256/0/180] [16658.443867] LNet: Added LNI 192.168.202.102@tcp126 [8/256/0/180] [16659.732493] LNet: Added LNI 192.168.202.102@tcp127 [8/256/0/180] [16660.887384] LNet: Added LNI 192.168.202.102@tcp128 [8/256/0/180] [16662.031471] LNet: Added LNI 192.168.202.102@tcp129 [8/256/0/180] [16663.156522] LNet: Added LNI 192.168.202.102@tcp130 [8/256/0/180] [16664.183373] LNet: Added LNI 192.168.202.102@tcp131 [8/256/0/180] [16665.022914] LNet: Added LNI 192.168.202.102@tcp132 [8/256/0/180] [16665.953872] LNet: Added LNI 192.168.202.102@tcp133 [8/256/0/180] [16666.913189] LNet: Added LNI 192.168.202.102@tcp134 [8/256/0/180] [16667.708378] LNet: Added LNI 192.168.202.102@tcp135 [8/256/0/180] [16674.696654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16674.722464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16675.941227] Lustre: 464691:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [16676.134293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16676.215069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16676.324969] Lustre: lustre-MDT0000: new disk, initializing [16676.422470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16676.445040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16681.637216] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16693.628902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16693.876527] Lustre: 464709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16693.975488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16693.979830] Lustre: Skipped 1 previous similar message [16694.071216] Lustre: lustre-MDT0001: new disk, initializing [16694.144486] Lustre: 465504:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [16694.153634] Lustre: 465504:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [16694.336229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16694.374486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16694.388850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16699.057253] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16701.516308] Lustre: 465869:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [16701.530545] Lustre: 465869:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [16710.663342] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16718.227337] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16726.034493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16726.347983] Lustre: lustre-OST0000: new disk, initializing [16726.355084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16726.366571] Lustre: 466681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16726.559540] Lustre: 466686:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [16726.576923] Lustre: 466686:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [16726.644189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16734.323849] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16736.287540] Lustre: 467137:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [16736.295360] Lustre: 467137:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [16736.376026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16736.384651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16736.474100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16744.788849] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16752.102939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16752.109971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16752.127359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16752.133335] Lustre: Skipped 1 previous similar message [16757.222112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16757.241330] Lustre: Skipped 1 previous similar message [16758.485771] Lustre: server umount lustre-OST0000 complete [16765.922244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16765.939711] LustreError: Skipped 1 previous similar message [16765.943448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16765.951970] Lustre: Skipped 1 previous similar message [16765.959825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16767.463507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16768.894318] Lustre: server umount lustre-MDT0000 complete [16773.424936] LustreError: 464700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477781 with bad export cookie 2660704631261338696 [16773.430672] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16773.443573] LustreError: 464700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16773.958860] Lustre: server umount lustre-MDT0001 complete [16787.888218] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16790.819580] Key type lgssc unregistered [16791.351338] LNet: 468597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16791.362449] LNetError: 468597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16792.415201] LNet: Removed LNI 192.168.202.102@tcp [16792.417147] LNetError: 468597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16792.429803] LNet: Removed LNI 192.168.202.102@tcp100 [16793.503165] LNet: Removed LNI 192.168.202.102@tcp101 [16793.505540] LNetError: 468597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16793.514704] LNetError: 468597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [16793.523116] LNet: Removed LNI 192.168.202.102@tcp102 [16793.533933] LNet: Removed LNI 192.168.202.102@tcp103 [16793.536145] LNet: Removed LNI 192.168.202.102@tcp104 [16793.540531] LNet: Removed LNI 192.168.202.102@tcp105 [16793.542383] LNet: Removed LNI 192.168.202.102@tcp106 [16793.552571] LNet: Removed LNI 192.168.202.102@tcp107 [16793.561993] LNet: Removed LNI 192.168.202.102@tcp108 [16793.574668] LNet: Removed LNI 192.168.202.102@tcp109 [16793.588222] LNet: Removed LNI 192.168.202.102@tcp110 [16793.593542] LNet: Removed LNI 192.168.202.102@tcp111 [16793.600556] LNet: Removed LNI 192.168.202.102@tcp112 [16793.605672] LNet: Removed LNI 192.168.202.102@tcp113 [16793.609444] LNet: Removed LNI 192.168.202.102@tcp114 [16793.612179] LNet: Removed LNI 192.168.202.102@tcp115 [16793.614579] LNet: Removed LNI 192.168.202.102@tcp116 [16793.620912] LNet: Removed LNI 192.168.202.102@tcp117 [16793.624215] LNet: Removed LNI 192.168.202.102@tcp118 [16793.627051] LNet: Removed LNI 192.168.202.102@tcp119 [16793.628944] LNet: Removed LNI 192.168.202.102@tcp120 [16793.637602] LNet: Removed LNI 192.168.202.102@tcp121 [16793.643350] LNet: Removed LNI 192.168.202.102@tcp122 [16793.645387] LNet: Removed LNI 192.168.202.102@tcp123 [16793.648209] LNet: Removed LNI 192.168.202.102@tcp124 [16793.655855] LNet: Removed LNI 192.168.202.102@tcp125 [16793.657944] LNet: Removed LNI 192.168.202.102@tcp126 [16793.660947] LNet: Removed LNI 192.168.202.102@tcp127 [16793.673654] LNet: Removed LNI 192.168.202.102@tcp128 [16793.676085] LNet: Removed LNI 192.168.202.102@tcp129 [16793.684767] LNet: Removed LNI 192.168.202.102@tcp130 [16793.686521] LNet: Removed LNI 192.168.202.102@tcp131 [16793.689347] LNet: Removed LNI 192.168.202.102@tcp132 [16793.692232] LNet: Removed LNI 192.168.202.102@tcp133 [16793.695127] LNet: Removed LNI 192.168.202.102@tcp134 [16793.708970] LNet: Removed LNI 192.168.202.102@tcp135 [16794.772198] Key type .llcrypt unregistered [16794.775545] Key type ._llcrypt unregistered [16821.075852] Key type ._llcrypt registered [16821.078445] Key type .llcrypt registered [16821.187261] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [16837.625717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16839.032548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16839.081590] alg: No test for adler32 (adler32-zlib) [16840.048805] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [16840.212314] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16841.858264] Key type lgssc registered [16842.871449] Lustre: Echo OBD driver; http://www.lustre.org/ [16881.933723] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:38:08 (1787477888) [16903.005953] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16905.720967] Key type lgssc unregistered [16906.074241] LNet: 472687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16906.080985] LNetError: 472687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16906.098620] LNet: Removed LNI 192.168.202.102@tcp [16907.203903] Key type .llcrypt unregistered [16907.208072] Key type ._llcrypt unregistered [16924.027282] Key type ._llcrypt registered [16924.030408] Key type .llcrypt registered [16924.160277] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16925.325323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16925.423544] alg: No test for adler32 (adler32-zlib) [16926.453665] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [16926.774345] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16928.408807] Key type lgssc registered [16929.499792] Lustre: Echo OBD driver; http://www.lustre.org/ [16941.458281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16941.501626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16942.744500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16942.779538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16942.881539] Lustre: lustre-MDT0000: new disk, initializing [16942.955212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16942.968275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16947.374206] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16957.975418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16958.052722] Lustre: 473909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16958.174262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16958.179355] Lustre: Skipped 1 previous similar message [16958.277387] Lustre: lustre-MDT0001: new disk, initializing [16958.423958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16958.512084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16958.537817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16962.668655] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16972.835375] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16979.405734] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16986.537461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16986.843862] Lustre: lustre-OST0000: new disk, initializing [16986.847712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16986.861295] Lustre: 475875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16986.990718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16993.733535] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16995.419410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16995.440479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16995.569675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17004.455951] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17008.683347] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lnet_if_list [17013.027864] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [17013.057763] LNet: Added LNI 192.168.202.102@tcp10 [8/256/0/180] [17018.356267] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [17018.408525] LNet: Added LNI 192.168.202.102@tcp11 [8/256/0/180] [17023.481686] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [17023.498732] LNet: Added LNI 192.168.202.102@tcp12 [8/256/0/180] [17028.033129] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [17028.071632] LNet: Added LNI 192.168.202.102@tcp13 [8/256/0/180] [17033.298567] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [17033.335652] LNet: Added LNI 192.168.202.102@tcp14 [8/256/0/180] [17038.597350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [17038.625616] LNet: Added LNI 192.168.202.102@tcp15 [8/256/0/180] [17043.547982] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [17043.582735] LNet: Added LNI 192.168.202.102@tcp16 [8/256/0/180] [17048.849600] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [17048.877403] LNet: Added LNI 192.168.202.102@tcp17 [8/256/0/180] [17053.869959] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [17053.904761] LNet: Added LNI 192.168.202.102@tcp18 [8/256/0/180] [17059.271352] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [17059.301432] LNet: Added LNI 192.168.202.102@tcp19 [8/256/0/180] [17066.307868] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [17066.355441] LNet: Added LNI 192.168.202.102@tcp20 [8/256/0/180] [17073.886263] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [17073.921741] LNet: Added LNI 192.168.202.102@tcp21 [8/256/0/180] [17081.472374] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [17081.515835] LNet: Added LNI 192.168.202.102@tcp22 [8/256/0/180] [17087.433263] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [17087.470228] LNet: Added LNI 192.168.202.102@tcp23 [8/256/0/180] [17094.172838] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [17094.192472] LNet: Added LNI 192.168.202.102@tcp24 [8/256/0/180] [17101.600952] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [17101.634841] LNet: Added LNI 192.168.202.102@tcp25 [8/256/0/180] [17111.620284] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [17111.700168] LNet: Added LNI 192.168.202.102@tcp26 [8/256/0/180] [17120.034447] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [17120.129292] LNet: Added LNI 192.168.202.102@tcp27 [8/256/0/180] [17129.141772] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [17129.170951] LNet: Added LNI 192.168.202.102@tcp28 [8/256/0/180] [17137.092869] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [17137.136743] LNet: Added LNI 192.168.202.102@tcp29 [8/256/0/180] [17144.587279] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [17144.601286] LNet: Added LNI 192.168.202.102@tcp30 [8/256/0/180] [17152.875459] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [17152.927619] LNet: Added LNI 192.168.202.102@tcp31 [8/256/0/180] [17160.101681] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [17160.133460] LNet: Added LNI 192.168.202.102@tcp32 [8/256/0/180] [17166.774573] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [17166.796594] LNet: Added LNI 192.168.202.102@tcp33 [8/256/0/180] [17173.560081] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [17173.577605] LNet: Added LNI 192.168.202.102@tcp34 [8/256/0/180] [17180.032475] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [17180.070612] LNet: Added LNI 192.168.202.102@tcp35 [8/256/0/180] [17186.653249] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [17186.700527] LNet: Added LNI 192.168.202.102@tcp36 [8/256/0/180] [17193.782501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [17193.807821] LNet: Added LNI 192.168.202.102@tcp37 [8/256/0/180] [17200.463385] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [17200.495829] LNet: Added LNI 192.168.202.102@tcp38 [8/256/0/180] [17207.832631] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [17207.872147] LNet: Added LNI 192.168.202.102@tcp39 [8/256/0/180] [17217.490665] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [17217.512810] LNet: Added LNI 192.168.202.102@tcp40 [8/256/0/180] [17227.400462] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [17227.443278] LNet: Added LNI 192.168.202.102@tcp41 [8/256/0/180] [17239.506957] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [17239.531106] LNet: Added LNI 192.168.202.102@tcp42 [8/256/0/180] [17247.652773] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [17247.678603] LNet: Added LNI 192.168.202.102@tcp43 [8/256/0/180] [17257.774234] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [17257.791881] LNet: Added LNI 192.168.202.102@tcp44 [8/256/0/180] [17264.528906] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [17264.554968] LNet: Added LNI 192.168.202.102@tcp45 [8/256/0/180] [17273.093873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [17273.155369] LNet: Added LNI 192.168.202.102@tcp46 [8/256/0/180] [17281.092467] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [17281.124033] LNet: Added LNI 192.168.202.102@tcp47 [8/256/0/180] [17291.872165] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [17291.897925] LNet: Added LNI 192.168.202.102@tcp48 [8/256/0/180] [17299.778877] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [17299.802430] LNet: Added LNI 192.168.202.102@tcp49 [8/256/0/180] [17308.996562] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [17309.027279] LNet: Added LNI 192.168.202.102@tcp50 [8/256/0/180] [17332.194788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17332.204182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17332.233302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17333.730480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17333.734142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17333.741558] Lustre: Skipped 1 previous similar message [17333.751592] Lustre: Skipped 2 previous similar messages [17336.575508] Lustre: server umount lustre-MDT0000 complete [17341.125822] LustreError: 473900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478349 with bad export cookie 7605824718366054336 [17341.135552] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17341.155695] LustreError: 473900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17341.969798] Lustre: server umount lustre-MDT0001 complete [17360.095182] Lustre: 473182:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787478352/real 1787478352] req@0000000080ff7395 x1874306462307200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787478368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17360.130400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17363.423207] Lustre: 473184:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787478355/real 1787478355] req@0000000030ed4334 x1874306462307328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787478371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17364.493682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [17365.478328] Lustre: 473184:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787478357/real 1787478357] req@00000000c8d230e1 x1874306462307456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787478373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17367.521216] Lustre: 473181:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787478359/real 1787478359] req@000000008e92aa8a x1874306462307712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787478375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17372.704110] Lustre: 473181:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787478364/real 1787478364] req@00000000fb3e5845 x1874306462307840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787478380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17374.582640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17386.784669] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x698d5148b218227f [17386.801583] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [17387.114665] LustreError: 475868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17387.147301] LustreError: 475868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [17387.171072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17389.222739] LustreError: 475869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17389.227920] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [17390.241201] LustreError: 475868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17391.485415] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17395.701678] LustreError: 475869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17399.304035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17399.609785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17403.506436] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17405.926446] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [17411.573443] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17417.940556] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17446.885710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17446.890252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17446.906302] Lustre: Skipped 1 previous similar message [17446.921782] Lustre: Skipped 1 previous similar message [17452.006075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17452.012083] Lustre: Skipped 1 previous similar message [17452.501449] Lustre: server umount lustre-OST0000 complete [17460.196738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17460.197863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17460.217201] Lustre: Skipped 1 previous similar message [17460.236793] Lustre: Skipped 1 previous similar message [17462.790821] Lustre: server umount lustre-MDT0000 complete [17465.312097] LustreError: 500130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17465.334445] LustreError: 500130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [17466.843705] LustreError: 473900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478474 with bad export cookie 7605824718366057087 [17466.848940] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17466.854619] LustreError: 473900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17467.115290] Lustre: server umount lustre-MDT0001 complete [17479.477485] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [17482.248496] Key type lgssc unregistered [17482.623948] LNet: 502930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17482.630520] LNetError: 502930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17483.682020] LNet: Removed LNI 192.168.202.102@tcp [17483.686959] LNetError: 502930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17483.692346] LNet: Removed LNI 192.168.202.102@tcp10 [17483.700255] LNet: Removed LNI 192.168.202.102@tcp11 [17483.703158] LNet: Removed LNI 192.168.202.102@tcp12 [17483.709421] LNet: Removed LNI 192.168.202.102@tcp13 [17483.714577] LNet: Removed LNI 192.168.202.102@tcp14 [17483.717288] LNet: Removed LNI 192.168.202.102@tcp15 [17483.719172] LNet: Removed LNI 192.168.202.102@tcp16 [17483.722365] LNet: Removed LNI 192.168.202.102@tcp17 [17483.728624] LNet: Removed LNI 192.168.202.102@tcp18 [17483.740224] LNet: Removed LNI 192.168.202.102@tcp19 [17483.746513] LNet: Removed LNI 192.168.202.102@tcp20 [17483.749945] LNet: Removed LNI 192.168.202.102@tcp21 [17483.754287] LNet: Removed LNI 192.168.202.102@tcp22 [17483.758556] LNet: Removed LNI 192.168.202.102@tcp23 [17483.771083] LNet: Removed LNI 192.168.202.102@tcp24 [17483.779738] LNet: Removed LNI 192.168.202.102@tcp25 [17483.786266] LNet: Removed LNI 192.168.202.102@tcp26 [17483.788220] LNet: Removed LNI 192.168.202.102@tcp27 [17483.793539] LNet: Removed LNI 192.168.202.102@tcp28 [17483.801957] LNet: Removed LNI 192.168.202.102@tcp29 [17483.805527] LNet: Removed LNI 192.168.202.102@tcp30 [17483.808562] LNet: Removed LNI 192.168.202.102@tcp31 [17483.812861] LNet: Removed LNI 192.168.202.102@tcp32 [17483.819953] LNet: Removed LNI 192.168.202.102@tcp33 [17483.831970] LNet: Removed LNI 192.168.202.102@tcp34 [17483.837213] LNet: Removed LNI 192.168.202.102@tcp35 [17483.842519] LNet: Removed LNI 192.168.202.102@tcp36 [17483.846853] LNet: Removed LNI 192.168.202.102@tcp37 [17483.850674] LNet: Removed LNI 192.168.202.102@tcp38 [17483.853277] LNet: Removed LNI 192.168.202.102@tcp39 [17483.856997] LNet: Removed LNI 192.168.202.102@tcp40 [17483.859045] LNet: Removed LNI 192.168.202.102@tcp41 [17483.862695] LNet: Removed LNI 192.168.202.102@tcp42 [17483.864559] LNet: Removed LNI 192.168.202.102@tcp43 [17483.867612] LNet: Removed LNI 192.168.202.102@tcp44 [17483.870179] LNet: Removed LNI 192.168.202.102@tcp45 [17483.872979] LNet: Removed LNI 192.168.202.102@tcp46 [17483.875724] LNet: Removed LNI 192.168.202.102@tcp47 [17483.877434] LNet: Removed LNI 192.168.202.102@tcp48 [17483.881631] LNet: Removed LNI 192.168.202.102@tcp49 [17483.897362] LNet: Removed LNI 192.168.202.102@tcp50 [17484.667956] Key type .llcrypt unregistered [17484.670144] Key type ._llcrypt unregistered [17504.711830] Key type ._llcrypt registered [17504.713494] Key type .llcrypt registered [17504.799383] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [17521.554486] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [17523.350948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17523.391728] alg: No test for adler32 (adler32-zlib) [17524.488258] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [17524.691092] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [17526.360560] Key type lgssc registered [17527.534535] Lustre: Echo OBD driver; http://www.lustre.org/ [17560.276658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17560.307498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17561.573240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17561.591417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17561.635976] Lustre: lustre-MDT0000: new disk, initializing [17561.706124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17561.736760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17566.253278] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17577.055297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17577.129542] Lustre: 506129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17577.178529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17577.181932] Lustre: Skipped 1 previous similar message [17577.242711] Lustre: lustre-MDT0001: new disk, initializing [17577.327161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17577.366354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17577.377062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17581.515160] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17589.887990] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17594.957642] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17602.670783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17602.860070] Lustre: *** cfs_fail_loc=90f, val=0*** [17602.950648] Lustre: lustre-OST0000: new disk, initializing [17602.954289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17602.961798] Lustre: 508147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17603.025840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17609.291610] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17609.795033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17609.809270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17609.943549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17618.920608] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17630.688520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17630.689289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17630.698509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17630.703091] Lustre: Skipped 1 previous similar message [17634.906937] Lustre: server umount lustre-OST0000 complete [17640.930743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17641.013092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17641.023394] Lustre: Skipped 2 previous similar messages [17646.278684] Lustre: server umount lustre-MDT0000 complete [17651.716949] LustreError: 506120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478659 with bad export cookie 17998787056925302623 [17651.728062] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17651.734373] LustreError: 506120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17652.388080] Lustre: server umount lustre-MDT0001 complete [17666.338846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [17668.674894] Key type lgssc unregistered [17668.970341] LNet: 510055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17668.975986] LNetError: 510055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17668.994222] LNet: Removed LNI 192.168.202.102@tcp [17670.225240] Key type .llcrypt unregistered [17670.231077] Key type ._llcrypt unregistered [17690.609407] Key type ._llcrypt registered [17690.613461] Key type .llcrypt registered [17690.764153] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [17691.938764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17692.007512] alg: No test for adler32 (adler32-zlib) [17693.013848] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [17693.189327] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [17694.831185] Key type lgssc registered [17695.798863] Lustre: Echo OBD driver; http://www.lustre.org/ [17703.235859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17703.255348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17704.598073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17708.790410] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17714.159923] LustreError: 511327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17714.191441] LustreError: 511327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [17718.991633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17719.404207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17725.877342] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17735.215267] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17741.964613] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17749.494851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17749.817544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17756.328753] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17764.841876] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17786.416337] loop0: detected capacity change from 0 to 204800000 [17786.626415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17786.643733] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17786.654748] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17792.817199] loop0: detected capacity change from 0 to 204800000 [17793.082607] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17793.098704] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17793.108127] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17801.443711] loop0: detected capacity change from 0 to 204800000 [17809.352149] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17809.717705] Lustre: fs15246-OST0000: new disk, initializing [17809.726935] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [17809.742735] Lustre: 514942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17809.968281] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17817.572558] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17828.472765] loop1: detected capacity change from 0 to 204800000 [17834.535873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17834.639944] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [17834.747853] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [17834.894667] Lustre: fs15246-MDT0000: new disk, initializing [17835.161295] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17835.203977] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17835.384541] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [17836.456941] Lustre: 516063:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000bebc98e x1874307266432640/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787478860 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [17840.218441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [17845.606061] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17845.617540] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17845.659052] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17870.311350] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17870.339037] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17872.604272] Lustre: server umount fs15246-MDT0000 complete [17875.425951] LustreError: 511324:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17880.566234] LustreError: 512424:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17883.044313] Lustre: server umount fs15246-OST0000 complete [17895.905662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17895.908151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17895.920131] Lustre: Skipped 1 previous similar message [17895.942021] Lustre: Skipped 1 previous similar message [17901.028587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17901.048373] Lustre: Skipped 1 previous similar message [17906.148913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17910.239196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17910.412706] Lustre: server umount lustre-OST0000 complete [17916.387116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17916.393532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17916.402176] Lustre: Skipped 1 previous similar message [17916.412259] Lustre: Skipped 1 previous similar message [17920.275538] Lustre: server umount lustre-MDT0000 complete [17921.514786] LustreError: 511328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17921.550894] LustreError: 511328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [17924.743408] LustreError: 517204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478932 with bad export cookie 4958754568737966557 [17924.745894] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17924.750719] LustreError: 517204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17925.394540] Lustre: server umount lustre-MDT0001 complete [17937.295830] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [17940.333985] Key type lgssc unregistered [17940.627891] LNet: 519168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17940.636859] LNetError: 519168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17940.659315] LNet: Removed LNI 192.168.202.102@tcp [17941.747973] Key type .llcrypt unregistered [17941.750724] Key type ._llcrypt unregistered [17961.184756] Key type ._llcrypt registered [17961.187314] Key type .llcrypt registered [17961.301517] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [17962.389252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17962.509933] alg: No test for adler32 (adler32-zlib) [17963.553919] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [17963.763347] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [17965.431179] Key type lgssc registered [17966.539978] Lustre: Echo OBD driver; http://www.lustre.org/ [18005.470957] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [18015.102777] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [18053.059589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18053.105258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18054.523528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18054.567686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18054.711911] Lustre: lustre-MDT0000: new disk, initializing [18054.804782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18054.845438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18059.377835] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [18072.237160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18072.385953] Lustre: 523636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18072.476841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [18072.477664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18072.493202] Lustre: Skipped 1 previous similar message [18072.594461] Lustre: lustre-MDT0001: new disk, initializing [18072.708379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18072.755672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18072.775238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18079.094730] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [18089.816202] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18095.722720] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18102.590397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18102.791949] Lustre: lustre-OST0000: new disk, initializing [18102.795263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18102.801266] Lustre: 525610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18102.900386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18108.462314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18108.483983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18108.583291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18109.698718] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [18119.512892] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18124.079320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [18124.286413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [18129.171890] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [18129.396539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [18134.497141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18134.499726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18134.519733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18134.528081] LustreError: Skipped 1 previous similar message [18134.542240] Lustre: Skipped 1 previous similar message [18137.608734] Lustre: server umount lustre-OST0000 complete [18144.224522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18144.236307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18144.250793] Lustre: Skipped 1 previous similar message [18144.255743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18147.067477] Lustre: server umount lustre-MDT0000 complete [18150.621121] LustreError: 524436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479158 with bad export cookie 12041319337807239987 [18150.628352] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18150.919701] Lustre: server umount lustre-MDT0001 complete [18177.950650] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [18189.102652] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [18221.191293] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17919 sec ========= 06:00:28 (1787479228) [18223.505229] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:00:30 (1787479230) === [18227.465781] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:00:35 (1787479235) === [18269.307830] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [18272.588124] Key type lgssc unregistered [18272.987714] LNet: 532269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18272.992947] LNetError: 532269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18273.011806] LNet: Removed LNI 192.168.202.102@tcp [18274.016855] Key type .llcrypt unregistered [18274.024455] Key type ._llcrypt unregistered