[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 531649658 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002417] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.007974] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011112] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014102] *** VALIDATE selinux *** [ 0.023497] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028978] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033129] *** VALIDATE tmpfs *** [ 0.035508] *** VALIDATE proc *** [ 0.037016] *** VALIDATE cgroup *** [ 0.038010] *** VALIDATE cgroup2 *** [ 0.039284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041083] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043036] Spectre V2 : User space: Vulnerable [ 0.044009] Speculative Store Bypass: Vulnerable [ 0.046710] debug: unmapping init [mem 0xffffffffbd259000-0xffffffffbd260fff] [ 0.048179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050024] ... version: 2 [ 0.050990] ... bit width: 48 [ 0.051017] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056287] rcu: Hierarchical SRCU implementation. [ 0.058527] smp: Bringing up secondary CPUs ... [ 0.059623] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.063682] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146285] node 0 deferred pages initialised in 78ms [ 0.150197] devtmpfs: initialized [ 0.151241] x86/mm: Memory block size: 128MB [ 0.153819] gcov: version magic: 0x41383552 [ 0.155169] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157305] pinctrl core: initialized pinctrl subsystem [ 0.158249] [ 0.158879] ************************************************************* [ 0.159017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160017] ** ** [ 0.161013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162016] ** ** [ 0.163015] ** This means that this kernel is built to expose internal ** [ 0.164015] ** IOMMU data structures, which may compromise security on ** [ 0.165017] ** your system. ** [ 0.166013] ** ** [ 0.167017] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169017] ** ** [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ************************************************************* [ 0.172656] NET: Registered protocol family 16 [ 0.173481] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176494] cpuidle: using governor menu [ 0.179236] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183581] PCI: Using configuration type 1 for base access [ 0.186133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199086] cryptd: max_cpu_qlen set to 1000 [ 0.201248] ACPI: Added _OSI(Module Device) [ 0.202015] ACPI: Added _OSI(Processor Device) [ 0.203016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208135] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214313] ACPI: Interpreter enabled [ 0.216071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218015] ACPI: Using IOAPIC for interrupt routing [ 0.219126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234076] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247400] acpiphp: Slot [2] registered [ 0.249099] acpiphp: Slot [5] registered [ 0.250100] acpiphp: Slot [6] registered [ 0.251139] acpiphp: Slot [7] registered [ 0.253128] acpiphp: Slot [8] registered [ 0.254098] acpiphp: Slot [9] registered [ 0.255126] acpiphp: Slot [10] registered [ 0.257108] acpiphp: Slot [3] registered [ 0.258122] acpiphp: Slot [4] registered [ 0.260098] acpiphp: Slot [11] registered [ 0.261107] acpiphp: Slot [12] registered [ 0.262088] acpiphp: Slot [13] registered [ 0.264092] acpiphp: Slot [14] registered [ 0.265143] acpiphp: Slot [15] registered [ 0.266098] acpiphp: Slot [16] registered [ 0.268145] acpiphp: Slot [17] registered [ 0.270110] acpiphp: Slot [18] registered [ 0.271099] acpiphp: Slot [19] registered [ 0.272082] acpiphp: Slot [20] registered [ 0.274080] acpiphp: Slot [21] registered [ 0.275090] acpiphp: Slot [22] registered [ 0.276131] acpiphp: Slot [23] registered [ 0.278100] acpiphp: Slot [24] registered [ 0.279087] acpiphp: Slot [25] registered [ 0.281077] acpiphp: Slot [26] registered [ 0.282084] acpiphp: Slot [27] registered [ 0.283069] acpiphp: Slot [28] registered [ 0.284079] acpiphp: Slot [29] registered [ 0.285100] acpiphp: Slot [30] registered [ 0.286085] acpiphp: Slot [31] registered [ 0.287080] PCI host bridge to bus 0000:00 [ 0.289018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304840] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307989] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314731] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330701] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334081] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.340011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.360017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368844] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428155] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.443029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480667] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.494021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.519017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.532784] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579780] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.593021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.626017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.637458] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658042] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.691042] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692306] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694200] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.696254] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701124] iommu: Default domain type: Passthrough [ 0.702241] SCSI subsystem initialized [ 0.703000] ACPI: bus type USB registered [ 0.704067] usbcore: registered new interface driver usbfs [ 0.706102] usbcore: registered new interface driver hub [ 0.707125] usbcore: registered new device driver usb [ 0.709165] pps_core: LinuxPPS API ver. 1 registered [ 0.710008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.712062] PTP clock support registered [ 0.714085] EDAC MC: Ver: 3.0.0 [ 0.717029] PCI: Using ACPI for IRQ routing [ 0.718973] NetLabel: Initializing [ 0.719012] NetLabel: domain hash size = 128 [ 0.720009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.721149] NetLabel: unlabeled traffic allowed by default [ 0.723211] vgaarb: loaded [ 0.725224] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.734272] clocksource: Switched to clocksource kvm-clock [ 0.840312] VFS: Disk quotas dquot_6.6.0 [ 0.841848] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.844026] *** VALIDATE ramfs *** [ 0.845027] *** VALIDATE hugetlbfs *** [ 0.846418] pnp: PnP ACPI init [ 0.848152] pnp: PnP ACPI: found 6 devices [ 0.864043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.867394] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.869715] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.871969] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.874236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.876603] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.879188] NET: Registered protocol family 2 [ 0.881471] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884869] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888119] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893522] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.896708] TCP: Hash tables configured (established 65536 bind 65536) [ 0.899563] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.902748] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.905525] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908271] NET: Registered protocol family 1 [ 0.910727] RPC: Registered named UNIX socket transport module. [ 0.912651] RPC: Registered udp transport module. [ 0.914108] RPC: Registered tcp transport module. [ 0.915573] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917629] NET: Registered protocol family 44 [ 0.919232] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.920678] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.922246] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.923774] PCI: CLS 0 bytes, default 64 [ 0.924896] Unpacking initramfs... [ 2.331490] debug: unmapping init [mem 0xffff95c97cc54000-0xffff95c97ffbffff] [ 2.335206] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.337136] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.339519] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.831300] Initialise system trusted keyrings [ 2.832684] Key type blacklist registered [ 2.834321] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.842313] zbud: loaded [ 2.844742] *** VALIDATE nfs *** [ 2.845962] *** VALIDATE nfs4 *** [ 2.847477] pstore: using deflate compression [ 2.850917] Platform Keyring initialized [ 2.951382] NET: Registered protocol family 38 [ 2.952853] Key type asymmetric registered [ 2.954190] Asymmetric key parser 'x509' registered [ 2.955871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.958498] io scheduler mq-deadline registered [ 2.959926] io scheduler kyber registered [ 2.961801] io scheduler bfq registered [ 2.963498] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.966862] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.969405] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.971880] ACPI: Power Button [PWRF] [ 2.976688] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.982091] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.996735] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.003259] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.021307] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.049179] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.078136] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.082186] Non-volatile memory driver v1.3 [ 3.083832] Linux agpgart interface v0.103 [ 3.113429] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.116413] vda: detected capacity change from 0 to 75100160 [ 3.128440] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.130491] vdb: detected capacity change from 0 to 1073741824 [ 3.142672] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.145092] vdc: detected capacity change from 0 to 2621440000 [ 3.161580] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.163796] vdd: detected capacity change from 0 to 2621440000 [ 3.182141] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.185370] vde: detected capacity change from 0 to 4294967296 [ 3.200569] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203103] vdf: detected capacity change from 0 to 4294967296 [ 3.209474] libphy: Fixed MDIO Bus: probed [ 3.216450] usbcore: registered new interface driver usbserial_generic [ 3.218555] usbserial: USB Serial support registered for generic [ 3.220380] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.223550] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.225438] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.227944] mousedev: PS/2 mouse device common for all mice [ 3.230275] rtc_cmos 00:05: RTC can wake from S4 [ 3.232870] rtc_cmos 00:05: registered as rtc0 [ 3.235438] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.235624] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.238183] intel_pstate: CPU model not supported [ 3.244123] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.246732] hid: raw HID events driver (C) Jiri Kosina [ 3.249438] usbcore: registered new interface driver usbhid [ 3.249540] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.251365] usbhid: USB HID core driver [ 3.251548] drop_monitor: Initializing network drop monitor service [ 3.257827] Initializing XFRM netlink socket [ 3.259701] NET: Registered protocol family 10 [ 3.262806] Segment Routing with IPv6 [ 3.264350] NET: Registered protocol family 17 [ 3.266442] mpls_gso: MPLS GSO support [ 3.272322] RAS: Correctable Errors collector initialized. [ 3.274312] AVX version of gcm_enc/dec engaged. [ 3.275854] AES CTR mode by8 optimization enabled [ 3.341290] sched_clock: Marking stable (3341266900, 0)->(4313311074, -972044174) [ 3.343501] registered taskstats version 1 [ 3.345252] Loading compiled-in X.509 certificates [ 3.347867] zswap: loaded using pool lzo/zbud [ 3.370438] Key type big_key registered [ 3.381668] Key type encrypted registered [ 3.383585] ima: No TPM chip found, activating TPM-bypass! [ 3.385819] ima: Allocated hash algorithm: sha1 [ 3.387826] ima: No architecture policies found [ 3.389772] evm: Initialising EVM extended attributes: [ 3.391827] evm: security.selinux [ 3.393395] evm: security.ima [ 3.394582] evm: security.capability [ 3.396047] evm: HMAC attrs: 0x1 [ 3.398708] rtc_cmos 00:05: setting system clock to 2026-09-13 11:08:20 UTC (1789297700) [ 3.405761] debug: unmapping init [mem 0xffffffffbe203000-0xffffffffbe3fffff] [ 3.409074] debug: unmapping init [mem 0xffffffffbcf82000-0xffffffffbd258fff] [ 3.422219] Write protecting the kernel read-only data: 28672k [ 3.425958] debug: unmapping init [mem 0xffffffffbb603000-0xffffffffbb7fffff] [ 3.429099] debug: unmapping init [mem 0xffffffffbbf14000-0xffffffffbbffffff] [ 3.466820] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.476167] systemd[1]: Detected virtualization kvm. [ 3.478261] systemd[1]: Detected architecture x86-64. [ 3.480343] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.506340] systemd[1]: No hostname configured. [ 3.508320] systemd[1]: Set hostname to . [ 3.510334] random: systemd: uninitialized urandom read (16 bytes read) [ 3.513149] systemd[1]: Initializing machine ID from random generator. [ 3.562812] random: ln: uninitialized urandom read (6 bytes read) [ 3.660393] random: systemd: uninitialized urandom read (16 bytes read) [ 3.663771] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.668440] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.674966] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.262462] device-mapper: uevent: version 1.0.3 [ 4.264488] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.037965] virtio_net virtio0 ens2: renamed from eth0 [ 5.065456] random: fast init done [ 5.242726] scsi host0: ata_piix [ 5.264232] scsi host1: ata_piix [ 5.265365] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.267125] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.656125] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.901658] random: crng init done [ 9.903254] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.335146] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.363237] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.579735] SELinux: Disabled at runtime. [ 11.643278] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.652553] systemd[1]: Detected virtualization kvm. [ 11.654276] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.129162] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.132490] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.136728] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.140947] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.144343] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.150822] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.155113] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ 12.214809] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.489756] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.817401] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.963590] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.065483] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.076308] EDAC sbridge: Ver: 1.1.2 [ 14.307711] Key type dns_resolver registered [ 14.608658] NFS: Registering the id_resolver key type [ 14.610449] Key type id_resolver registered [ 14.611869] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg255-server login: [ 40.245569] libcfs: loading out-of-tree module taints kernel. [ 40.265568] Key type ._llcrypt registered [ 40.267193] Key type .llcrypt registered [ 40.321513] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 48.474636] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 49.077674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.083821] alg: No test for adler32 (adler32-zlib) [ 50.140805] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 50.485846] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 52.112180] Key type lgssc registered [ 52.814246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 60.042124] hrtimer: interrupt took 19932617 ns [ 71.503658] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.813961] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 129.928728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.964882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.365367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.398476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.487577] Lustre: lustre-MDT0000: new disk, initializing [ 131.565919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.582566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.041843] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.366981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.523863] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.579362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.591396] Lustre: Skipped 1 previous similar message [ 149.760307] Lustre: lustre-MDT0001: new disk, initializing [ 149.874844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.929086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.940316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.550259] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.880547] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.418075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.897280] Lustre: lustre-OST0000: new disk, initializing [ 171.901703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.909659] Lustre: 8420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.037604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.869727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.884972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.032507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 178.354474] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.819482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.022828] Lustre: lustre-OST0001: new disk, initializing [ 194.029952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.041067] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.133694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.211436] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.794278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 202.800843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.836821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.054656] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.755533] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.786558] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.769356] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing yml_node [ 238.344277] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 241.945151] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 244.718380] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 246.914766] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 07:12:23 EDT 2026 [ 264.382967] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 266.102595] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 283.105462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 283.118884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.143382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.641652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.641975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.649691] Lustre: Skipped 1 previous similar message [ 284.679557] Lustre: Skipped 1 previous similar message [ 288.688260] Lustre: server umount lustre-MDT0000 complete [ 293.153163] LustreError: 8430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297990 with bad export cookie 7826783250209938233 [ 293.154798] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.177716] LustreError: 8430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 294.881105] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.887379] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.890978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.890987] Lustre: Skipped 2 previous similar messages [ 294.903013] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 294.951393] Lustre: Skipped 2 previous similar messages [ 298.913677] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.936257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.946296] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 298.977779] Lustre: Skipped 2 previous similar messages [ 300.021291] Lustre: server umount lustre-MDT0001 complete [ 303.656813] Lustre: server umount lustre-OST0000 complete [ 308.593566] Lustre: server umount lustre-OST0001 complete [ 316.134655] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 325.405874] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 361.997382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.339869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.369889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.471219] Lustre: lustre-MDT0000: new disk, initializing [ 362.542946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.556659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.724239] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 378.945598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.091284] Lustre: 15049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.143981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.147839] Lustre: Skipped 1 previous similar message [ 379.204529] Lustre: lustre-MDT0001: new disk, initializing [ 379.314317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.364631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.386891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.064351] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 393.969335] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.780819] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.246617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.454885] Lustre: lustre-OST0000: new disk, initializing [ 406.457123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.461094] Lustre: 17020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.536706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.619570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.628926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 407.720066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.525522] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 422.445360] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.624752] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.766645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.727524] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.964429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 438.758718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.768842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.770290] LustreError: Skipped 1 previous similar message [ 438.772831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.795319] Lustre: Skipped 1 previous similar message [ 440.139404] Lustre: server umount lustre-OST0000 complete [ 443.874556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.890059] Lustre: Skipped 1 previous similar message [ 448.994868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.007107] Lustre: Skipped 4 previous similar messages [ 449.936075] Lustre: server umount lustre-MDT0000 complete [ 454.077147] LustreError: 17019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298151 with bad export cookie 7826783250209939766 [ 454.080228] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.106045] LustreError: 17019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 454.609398] Lustre: server umount lustre-MDT0001 complete [ 459.545382] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:15:55 (1789298155) [ 462.065954] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 463.765344] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:16:00 (1789298160) [ 470.439307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.826665] LustreError: 19428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.901783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.096825] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 476.136102] LustreError: 19429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.230689] LustreError: 19428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.782947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.137785] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 499.595584] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.081915] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.902410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.916124] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 527.931368] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.153816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.166696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.205591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.202629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.036852] Lustre: server umount lustre-OST0000 complete [ 552.930944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.196118] Lustre: server umount lustre-MDT0000 complete [ 558.567363] LustreError: 19423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.591470] LustreError: 19423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 559.379598] LustreError: 19409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298256 with bad export cookie 7826783250209940648 [ 559.397262] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.399920] LustreError: 19409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 559.885095] Lustre: server umount lustre-MDT0001 complete [ 570.978827] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 573.721460] Key type lgssc unregistered [ 574.241235] LNet: 22960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.252493] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.284891] LNet: Removed LNI 192.168.202.155@tcp [ 575.787158] Key type .llcrypt unregistered [ 575.796900] Key type ._llcrypt unregistered [ 600.659630] Key type ._llcrypt registered [ 600.666868] Key type .llcrypt registered [ 600.841744] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 602.419846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.481072] alg: No test for adler32 (adler32-zlib) [ 603.566631] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 603.888589] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 605.576838] Key type lgssc registered [ 606.743682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 616.561965] loop: module loaded [ 616.669052] loop0: detected capacity change from 0 to 204800000 [ 616.936300] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 616.945098] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 617.175332] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.873972] loop0: detected capacity change from 0 to 204800000 [ 624.069174] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 624.083587] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 624.106231] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 629.752512] loop0: detected capacity change from 0 to 204800000 [ 635.734997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 635.776099] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.084945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 637.158937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 637.267337] Lustre: lustre-MDT0000: new disk, initializing [ 637.321089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.357926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 641.934253] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 652.054366] loop1: detected capacity change from 0 to 204800000 [ 658.838944] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.119695] Lustre: lustre-OST07c6: new disk, initializing [ 659.126836] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 659.137870] Lustre: Skipped 1 previous similar message [ 659.148336] Lustre: 26147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 659.219275] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 665.800724] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 669.260235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 669.267119] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 669.318640] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 671.764876] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 671.772577] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 671.777918] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 671.784872] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 671.790737] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 671.800874] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 687.309215] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 687.321193] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 687.330150] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.341059] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 687.351133] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 687.364574] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.405967] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 692.434290] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 692.455233] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 692.466415] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.473610] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 692.479033] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.482261] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 692.485571] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.489390] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 692.494378] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.497386] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.500701] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.774787] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 695.780896] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 695.789832] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 695.795641] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.804746] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.815953] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.828554] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 695.844909] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.857345] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 695.868781] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.885185] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 695.896778] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.234356] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 701.249720] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 701.261882] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 701.275811] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.287579] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.294668] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.304741] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 701.312245] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.323938] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.329159] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.341345] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.348734] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.587200] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 709.598872] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 709.607287] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.614199] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 709.623306] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.629054] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 709.649849] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 709.664798] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 709.673828] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 709.679522] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 709.688137] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.695589] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 725.907045] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 725.913660] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 725.923281] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.933635] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 725.939486] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.951923] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 725.962076] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.972527] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 725.981221] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.988424] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 726.000140] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.008849] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 766.434053] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.451935] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 770.163123] Lustre: server umount lustre-OST07c6 complete [ 780.204611] Lustre: server umount lustre-MDT0000 complete [ 800.001754] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 803.449697] Key type lgssc unregistered [ 803.710076] LNet: 28704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.727795] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.755201] LNet: Removed LNI 192.168.202.155@tcp [ 804.686688] Key type .llcrypt unregistered [ 804.689254] Key type ._llcrypt unregistered [ 824.675281] Key type ._llcrypt registered [ 824.677403] Key type .llcrypt registered [ 824.792367] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 825.958741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.126293] alg: No test for adler32 (adler32-zlib) [ 827.286075] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 827.618152] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 829.360197] Key type lgssc registered [ 830.827035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 839.007674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 839.035395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.603870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.912324] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 850.917737] LustreError: 29973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.940269] LustreError: 29973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 854.121366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.526326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 859.740870] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 868.253913] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.825700] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 883.790863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 884.225625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.281865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 890.678933] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 898.695476] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 916.962507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 916.976820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.991507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 917.768320] Lustre: server umount lustre-OST0000 complete [ 925.155292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.171484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.174474] Lustre: Skipped 1 previous similar message [ 925.187190] Lustre: Skipped 1 previous similar message [ 928.671450] Lustre: server umount lustre-MDT0000 complete [ 930.277937] LustreError: 30941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.311953] LustreError: 30941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 932.584458] LustreError: 32959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298629 with bad export cookie 2198950843465155103 [ 932.587740] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.600452] LustreError: 32959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 942.419671] Lustre: server umount lustre-MDT0001 complete [ 955.438864] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 958.989272] Key type lgssc unregistered [ 959.455766] LNet: 33711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.468248] LNetError: 33711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.490085] LNet: Removed LNI 192.168.202.155@tcp [ 961.344461] Key type .llcrypt unregistered [ 961.351423] Key type ._llcrypt unregistered [ 981.681779] Key type ._llcrypt registered [ 981.683487] Key type .llcrypt registered [ 981.786242] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 982.779706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.817433] alg: No test for adler32 (adler32-zlib) [ 983.943954] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 984.156828] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 985.826858] Key type lgssc registered [ 986.739803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.713418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.733641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.221074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.242512] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1005.553022] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.572272] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1007.460482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.971211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.576767] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1020.619131] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1025.773865] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.462608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.878579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.012505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1042.452473] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1050.926942] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1060.859091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1060.865946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.875912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.893206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.896517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1062.930726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.049242] Lustre: server umount lustre-OST0000 complete [ 1073.121971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.123947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.130145] Lustre: Skipped 1 previous similar message [ 1073.133498] Lustre: Skipped 1 previous similar message [ 1076.868825] Lustre: server umount lustre-MDT0000 complete [ 1078.241349] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.250768] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1079.999956] LustreError: 34958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298777 with bad export cookie 3112340294694969017 [ 1080.007911] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.012859] LustreError: 34958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1080.553398] Lustre: server umount lustre-MDT0001 complete [ 1092.180326] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1095.169563] Key type lgssc unregistered [ 1095.572815] LNet: 38511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.593728] LNetError: 38511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.607677] LNet: Removed LNI 192.168.202.155@tcp [ 1096.557862] Key type .llcrypt unregistered [ 1096.559933] Key type ._llcrypt unregistered [ 1117.490278] Key type ._llcrypt registered [ 1117.493106] Key type .llcrypt registered [ 1117.614825] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1118.443079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1118.457336] alg: No test for adler32 (adler32-zlib) [ 1119.537135] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1119.728518] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1121.376571] Key type lgssc registered [ 1122.489676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.279105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.298793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.642183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.361248] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1140.193897] LustreError: 39905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.212152] LustreError: 39905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1143.021889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.346070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.846752] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1157.388439] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.423818] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1172.656838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1173.190736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.241801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1179.442543] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1187.707158] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.963342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.966153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.990891] Lustre: Skipped 1 previous similar message [ 1194.012505] Lustre: Skipped 2 previous similar messages [ 1195.153273] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 1199.073862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.195205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.197789] Lustre: Skipped 5 previous similar messages [ 1208.288241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1208.513398] Lustre: server umount lustre-MDT0000 complete [ 1209.316461] LustreError: 39905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.328652] LustreError: 39905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1210.488285] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.043403] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298910 with bad export cookie 13125940790623696729 [ 1213.049645] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.067371] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1214.433993] LustreError: 39902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.434958] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.466386] Lustre: Skipped 1 previous similar message [ 1214.472089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1214.474878] Lustre: Skipped 1 previous similar message [ 1218.468231] LustreError: 39900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.483241] LustreError: 39900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1219.409700] Lustre: server umount lustre-MDT0001 complete [ 1234.757868] Lustre: server umount lustre-OST0000 complete [ 1252.331598] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1254.662935] Key type lgssc unregistered [ 1254.890171] LNet: 43763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.897704] LNetError: 43763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.914654] LNet: Removed LNI 192.168.202.155@tcp [ 1256.631979] Key type .llcrypt unregistered [ 1256.634057] Key type ._llcrypt unregistered [ 1280.891537] Key type ._llcrypt registered [ 1280.896035] Key type .llcrypt registered [ 1280.996867] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1282.248494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.279562] alg: No test for adler32 (adler32-zlib) [ 1283.288573] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1283.465274] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1285.081100] Key type lgssc registered [ 1286.190803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.898810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.918993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.471639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.067307] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1304.033157] LustreError: 45175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.049587] LustreError: 45175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1307.149707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.618252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.854916] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1319.516782] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1325.141791] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1332.857761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.244451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.421937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1340.978334] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1351.038789] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1357.807890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.809626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.856436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.859244] Lustre: Skipped 1 previous similar message [ 1357.878389] Lustre: Skipped 1 previous similar message [ 1358.482889] Lustre: lustre-OST0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 1362.913862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.919352] Lustre: Skipped 1 previous similar message [ 1368.035975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.057860] Lustre: Skipped 2 previous similar messages [ 1372.128155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1372.418069] Lustre: server umount lustre-OST0000 complete [ 1393.635470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.670274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.682079] Lustre: Skipped 2 previous similar messages [ 1396.837076] Lustre: server umount lustre-MDT0000 complete [ 1398.753288] LustreError: 46209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.790827] LustreError: 46209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1399.781578] LustreError: 47017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.805396] LustreError: 47017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1401.213251] LustreError: 45157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299098 with bad export cookie 11085048851197293424 [ 1401.219397] LustreError: 45157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1401.227608] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.019400] Lustre: server umount lustre-MDT0001 complete [ 1420.851505] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1424.108886] Key type lgssc unregistered [ 1424.541205] LNet: 48887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.556218] LNetError: 48887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.585474] LNet: Removed LNI 192.168.202.155@tcp [ 1425.563183] Key type .llcrypt unregistered [ 1425.565664] Key type ._llcrypt unregistered [ 1449.321439] Key type ._llcrypt registered [ 1449.323895] Key type .llcrypt registered [ 1449.445204] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1450.638714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1450.712751] alg: No test for adler32 (adler32-zlib) [ 1451.791986] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1452.022870] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1453.672195] Key type lgssc registered [ 1454.817738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.359555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1462.374851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.891991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.864639] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1473.008604] LustreError: 50298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.031912] LustreError: 50298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1478.117281] LustreError: 50299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.597319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.123252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.338897] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1495.273924] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.550986] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1510.699710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.054595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.885966] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1526.091147] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1533.624772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1543.112307] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1547.618301] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1560.420468] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-13 7h34m16s [ 1562.034929] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1564.647259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.658159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.666075] Lustre: Skipped 2 previous similar messages [ 1564.684975] Lustre: Skipped 2 previous similar messages [ 1565.734992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.745204] Lustre: Skipped 2 previous similar messages [ 1568.893701] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 1570.518976] Lustre: server umount lustre-MDT0000 complete [ 1571.818227] LustreError: 52741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.834420] LustreError: 52741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1574.403505] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1576.051362] LustreError: 50294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.917522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.052521] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.181984] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (not set up) [ 1582.192653] Lustre: Skipped 1 previous similar message [ 1582.378482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.472436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1583.417264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1587.196231] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1590.717497] Lustre: DEBUG MARKER: Wait for df (36647) ... [ 1592.455426] Lustre: DEBUG MARKER: done [ 1597.922344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1597.940733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.957337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.952338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.428880] Lustre: server umount lustre-OST0000 complete [ 1608.171364] LustreError: 49611:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000095df09b4 x1876216144526464/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1609.186714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.190974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.199168] Lustre: Skipped 1 previous similar message [ 1609.211083] Lustre: Skipped 3 previous similar messages [ 1614.739629] Lustre: server umount lustre-MDT0000 complete [ 1616.359600] LustreError: 50295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.386062] LustreError: 50295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1619.221215] LustreError: 50281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299316 with bad export cookie 14654302924337331769 [ 1619.231123] LustreError: 50281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1619.234492] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.560634] Lustre: server umount lustre-MDT0001 complete [ 1632.160928] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1635.248295] Key type lgssc unregistered [ 1635.559701] LNet: 55245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.564439] LNetError: 55245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.582609] LNet: Removed LNI 192.168.202.155@tcp [ 1636.459505] Key type .llcrypt unregistered [ 1636.460944] Key type ._llcrypt unregistered [ 1680.732492] Key type ._llcrypt registered [ 1680.734498] Key type .llcrypt registered [ 1680.887343] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1682.711264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.805201] alg: No test for adler32 (adler32-zlib) [ 1684.142692] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1684.591759] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1686.264189] Key type lgssc registered [ 1687.726047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.631813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.662365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.935299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.953959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1699.193502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.307761] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1715.832905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.914631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1715.937420] Lustre: 57453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1716.164078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1721.005540] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1728.574266] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1734.662642] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1744.062189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1744.253492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1744.432411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.884864] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1751.237156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1759.533992] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.385297] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-13 7h37m43s [ 1769.202547] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1776.496857] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1779.223971] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1779.305932] LustreError: 57459:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1799.847208] Lustre: lustre-MDT0000: Client 1d68fc1b-e3a9-4a35-988c-8cfe93cecbe6 (at 192.168.202.55@tcp) reconnecting [ 1819.369997] LustreError: 57459:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1822.152990] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1824.981699] Lustre: DEBUG MARKER: done [ 1832.929693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.937542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.949881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.444548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.458449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.482304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.495481] Lustre: Skipped 1 previous similar message [ 1837.450112] Lustre: server umount lustre-OST0000 complete [ 1843.685970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.691651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.718646] Lustre: Skipped 1 previous similar message [ 1843.739451] Lustre: Skipped 1 previous similar message [ 1848.029432] Lustre: server umount lustre-MDT0000 complete [ 1852.163641] LustreError: 57445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299549 with bad export cookie 4388909336233848422 [ 1852.173730] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1852.602828] Lustre: server umount lustre-MDT0001 complete [ 1864.487463] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1868.051912] Key type lgssc unregistered [ 1868.460915] LNet: 61782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.470140] LNetError: 61782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.491207] LNet: Removed LNI 192.168.202.155@tcp [ 1869.337120] Key type .llcrypt unregistered [ 1869.340085] Key type ._llcrypt unregistered [ 1923.547716] Key type ._llcrypt registered [ 1923.550606] Key type .llcrypt registered [ 1923.698869] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1925.040499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.276144] alg: No test for adler32 (adler32-zlib) [ 1926.587282] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 1927.095206] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1928.834065] Key type lgssc registered [ 1930.274968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.069844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.100842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.366800] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1935.389296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1935.539903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.539139] Lustre: Failing over lustre-MDT0000 [ 1936.872619] Lustre: server umount lustre-MDT0000 complete [ 1945.256799] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:40:42 (1789299642) [ 1951.836360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.414325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.811509] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1966.053517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.169094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1966.203972] Lustre: 64575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1966.505366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.103124] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1979.736447] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1985.179541] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1992.905156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.227775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1993.456273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.109950] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2002.596183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2010.162250] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.883744] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2028.003296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2028.026717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.074335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.634978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.642229] Lustre: Skipped 4 previous similar messages [ 2038.755277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.764583] Lustre: Skipped 2 previous similar messages [ 2041.312200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2041.487270] Lustre: server umount lustre-MDT0000 complete [ 2046.013051] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299743 with bad export cookie 9615063703739402343 [ 2046.028855] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2046.036949] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.977751] Lustre: server umount lustre-MDT0001 complete [ 2051.553874] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2056.430257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2065.128175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.312242] Lustre: 63499:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299746/real 1789299746] req@00000000bda57484 x1876216642579712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299762 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.333298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.343332] Lustre: Skipped 2 previous similar messages [ 2068.450181] Lustre: 63502:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299749/real 1789299749] req@00000000e0454fc7 x1876216642579840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.536328] Lustre: 63502:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299750/real 1789299750] req@000000006c2ddab7 x1876216642579968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.525156] LustreError: 63498:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e0454fc7 x1876216642580608/t0(0) o250->MGC192.168.202.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.901838] LustreError: 66426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.924041] LustreError: 66426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2071.993244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.109277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2072.993723] Lustre: 63499:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299753/real 1789299753] req@00000000226208c8 x1876216642580224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.099667] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2074.104046] LustreError: 66427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.179889] LustreError: 66426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.192209] LustreError: 66426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2076.666676] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2078.176861] Lustre: 63500:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299758/real 1789299758] req@000000005f9f84fc x1876216642580352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2080.228976] LustreError: 68351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.252571] LustreError: 68351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2085.356767] LustreError: 68349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.120115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.597737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.006252] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2093.046043] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.879174] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2107.649067] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2118.113555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2118.121848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.135776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.895101] Lustre: server umount lustre-MDT0000 complete [ 2123.749368] LustreError: 69085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.778638] LustreError: 69085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2125.107401] LustreError: 64568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299822 with bad export cookie 9615063703739404618 [ 2125.107941] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.133476] LustreError: 64568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2125.554594] Lustre: server umount lustre-MDT0001 complete [ 2133.120440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2144.230186] Lustre: 63501:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299825/real 1789299825] req@00000000b3756c13 x1876216642608640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.269853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.282272] Lustre: Skipped 2 previous similar messages [ 2145.556585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.109773] LustreError: 66427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.249743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.357745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2155.961587] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2165.849200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.211860] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2171.375420] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2173.479229] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2180.626977] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2188.629635] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2199.016652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.019582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.040889] Lustre: Skipped 2 previous similar messages [ 2199.043680] Lustre: Skipped 3 previous similar messages [ 2204.517108] Lustre: server umount lustre-MDT0000 complete [ 2208.828944] LustreError: 64566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299905 with bad export cookie 9615063703739405997 [ 2208.840076] LustreError: 64566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2208.842027] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.163902] Lustre: server umount lustre-MDT0001 complete [ 2215.666515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2219.436529] Lustre: DEBUG MARKER: files compared the same [ 2231.930992] Lustre: server umount lustre-OST0000 complete [ 2248.354895] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 2251.199561] Key type lgssc unregistered [ 2251.437761] LNet: 74309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.460353] LNetError: 74309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.493226] LNet: Removed LNI 192.168.202.155@tcp [ 2252.322789] Key type .llcrypt unregistered [ 2252.325890] Key type ._llcrypt unregistered [ 2277.309162] Key type ._llcrypt registered [ 2277.310973] Key type .llcrypt registered [ 2277.444374] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2278.435848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.502772] alg: No test for adler32 (adler32-zlib) [ 2279.679607] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 2280.004232] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 2281.720211] Key type lgssc registered [ 2282.999881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2290.825866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2290.839672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.365568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.098890] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug +malloc all [ 2301.921643] LustreError: 75722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.937837] LustreError: 75722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2305.276849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.637459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.993672] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug +malloc all [ 2318.869879] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.682434] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2333.123938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.391704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.523700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2341.137868] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug +malloc all [ 2349.332793] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2361.314028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.318690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.338687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2364.730415] Lustre: server umount lustre-OST0000 complete [ 2371.556993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.564874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.577470] Lustre: Skipped 2 previous similar messages [ 2371.592064] Lustre: Skipped 2 previous similar messages [ 2374.886438] Lustre: server umount lustre-MDT0000 complete [ 2376.677187] LustreError: 76450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.714861] LustreError: 76450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2378.701101] LustreError: 75702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300075 with bad export cookie 5991751668691774201 [ 2378.706423] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.723466] LustreError: 75702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2379.260131] Lustre: server umount lustre-MDT0001 complete [ 2391.403799] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 2395.349439] Key type lgssc unregistered [ 2395.855597] LNet: 79253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.877300] LNetError: 79253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.924088] LNet: Removed LNI 192.168.202.155@tcp [ 2396.918901] Key type .llcrypt unregistered [ 2396.920899] Key type ._llcrypt unregistered [ 2423.091898] Key type ._llcrypt registered [ 2423.093880] Key type .llcrypt registered [ 2423.189533] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2424.454602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2424.654870] alg: No test for adler32 (adler32-zlib) [ 2425.780585] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 2425.926034] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 2427.576436] Key type lgssc registered [ 2428.892930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2437.952988] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2437.976740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.536662] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2458.656965] LustreError: 80646:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.155@tcp: failed processing log, type 4: rc = -110 [ 2485.344293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.223611] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2499.009936] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2506.990129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.106086] Lustre: *** cfs_fail_loc=706, val=0*** [ 2507.544511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.620987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2512.218397] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2512.871247] LustreError: 80672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.886794] LustreError: 80672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2517.992856] LustreError: 80671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.888558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.041176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2520.059639] Lustre: Skipped 1 previous similar message [ 2520.235762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2524.443220] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2532.708108] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2539.122655] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2543.585952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2543.590690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.615096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.147050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.154997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.172288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.208703] Lustre: Skipped 1 previous similar message [ 2547.807918] Lustre: server umount lustre-OST0000 complete [ 2551.271437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.276472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2551.283161] LustreError: 79980:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000035a627e1 x1876217165798784/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2551.284382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.386845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.392631] Lustre: Skipped 2 previous similar messages [ 2557.628826] Lustre: server umount lustre-MDT0000 complete [ 2561.511941] LustreError: 81737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.530823] LustreError: 81737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2561.653452] LustreError: 80652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300258 with bad export cookie 12910638858168722514 [ 2561.660757] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.672278] LustreError: 80652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2562.062240] Lustre: server umount lustre-MDT0001 complete [ 2573.780157] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 2576.879388] Key type lgssc unregistered [ 2577.144040] LNet: 84260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2577.158066] LNetError: 84260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.191091] LNet: Removed LNI 192.168.202.155@tcp [ 2578.137643] Key type .llcrypt unregistered [ 2578.140128] Key type ._llcrypt unregistered [ 2605.604474] Key type ._llcrypt registered [ 2605.606765] Key type .llcrypt registered [ 2605.697194] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2606.731520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2606.929732] alg: No test for adler32 (adler32-zlib) [ 2608.026492] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 2608.309592] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 2610.033531] Key type lgssc registered [ 2611.074215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.827219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.852982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.327962] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2633.072733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.691620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2637.953750] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2638.837065] LustreError: 86361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.863049] LustreError: 86361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2642.916409] LustreError: 86364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.926021] LustreError: 86364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2647.732590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.036391] LustreError: 86365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.059239] LustreError: 86365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2648.218763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.670725] LustreError: 86361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.687270] LustreError: 86361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2654.463625] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2658.793849] LustreError: 86359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.813357] LustreError: 86359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2662.853993] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.872558] Lustre: MGS: Client 7ea540d8-0745-4f6d-8d3b-8877e4e00e3e (at 0@lo) reconnecting [ 2662.900612] LustreError: 86375:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2663.241321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.315389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2667.905842] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2683.883959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.909270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.930568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.943339] Lustre: Skipped 1 previous similar message [ 2688.998955] LustreError: 87089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.006546] LustreError: 87089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2689.050807] Lustre: server umount lustre-OST0000 complete [ 2694.113792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.133266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.134105] Lustre: Skipped 2 previous similar messages [ 2694.147539] Lustre: Skipped 1 previous similar message [ 2698.996789] Lustre: server umount lustre-MDT0000 complete [ 2703.006426] Lustre: server umount lustre-MDT0001 complete [ 2706.732624] Lustre: server umount MGS complete [ 2718.526995] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:53:35 (1789300415) [ 2753.096596] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 2763.748356] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2800.896385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.015951] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2817.794349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.112708] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2818.177728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2818.287760] Lustre: lustre-MDT0001: new disk, initializing [ 2818.396285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.421463] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2818.434928] Lustre: 93551:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2818.450056] LustreError: 93551:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2823.925101] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2836.180949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.579625] Lustre: lustre-OST0000: new disk, initializing [ 2836.584666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2836.592511] Lustre: 94318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2836.717681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.646591] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2846.770154] Lustre: 94775:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c02afbe1 x1876217356656256/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789300559 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2854.552119] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2861.942605] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.957556] Lustre: MGS: Client a3a0303d-2195-4c46-9129-f9b4677995f3 (at 0@lo) reconnecting [ 2862.016822] Lustre: 95365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.038393] LustreError: 93537:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2862.052490] LustreError: 93537:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2862.086257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2862.199953] Lustre: lustre-MDT0000: new disk, initializing [ 2862.420631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.480181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2862.505840] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.525206] Lustre: 95389:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a5414b14 x1876217356663168/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789300576 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2867.247570] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2868.769669] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2872.751366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2872.757608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2872.852662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2898.403515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2898.428881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.472597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.943710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.574440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.591914] Lustre: Skipped 3 previous similar messages [ 2910.690171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.700726] Lustre: Skipped 1 previous similar message [ 2911.200628] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2911.576524] Lustre: server umount lustre-OST0000 complete [ 2920.929411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.933382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2920.966339] Lustre: Skipped 1 previous similar message [ 2920.980891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.997516] Lustre: Skipped 2 previous similar messages [ 2922.792696] Lustre: server umount lustre-MDT0000 complete [ 2926.068069] LustreError: 93521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.100335] LustreError: 93521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2934.747939] Lustre: server umount lustre-MDT0001 complete [ 2939.093161] Lustre: server umount MGS complete [ 2953.714028] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:57:30 (1789300650) [ 2972.489326] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 2975.056246] Key type lgssc unregistered [ 2975.376120] LNet: 97888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2975.384784] LNetError: 97888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2975.410203] LNet: Removed LNI 192.168.202.155@tcp [ 2976.252763] Key type .llcrypt unregistered [ 2976.256188] Key type ._llcrypt unregistered [ 2996.062822] Key type ._llcrypt registered [ 2996.066972] Key type .llcrypt registered [ 2996.192625] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2997.177143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2997.246565] alg: No test for adler32 (adler32-zlib) [ 2998.361026] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 2998.604757] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3000.337546] Key type lgssc registered [ 3001.601877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.046539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3009.067391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.482805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.315269] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3020.777972] LustreError: 99108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.790444] LustreError: 99108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3024.023040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.389474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3029.003994] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3036.976348] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3042.562762] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3049.740504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3050.258701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.422822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3056.592062] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3064.313887] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3076.067370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3076.073860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.092698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.112191] Lustre: server umount lustre-OST0000 complete [ 3085.282342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.289161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.297967] Lustre: Skipped 1 previous similar message [ 3088.487885] Lustre: server umount lustre-MDT0000 complete [ 3090.403335] LustreError: 99107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.424516] LustreError: 99107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3092.663241] LustreError: 99088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300789 with bad export cookie 7022226080842232121 [ 3092.667791] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.675869] LustreError: 99088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3102.421767] Lustre: server umount lustre-MDT0001 complete [ 3113.252731] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3115.857263] Key type lgssc unregistered [ 3116.213784] LNet: 102639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3116.220104] LNetError: 102639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3116.235596] LNet: Removed LNI 192.168.202.155@tcp [ 3117.084264] Key type .llcrypt unregistered [ 3117.086614] Key type ._llcrypt unregistered [ 3136.311418] Key type ._llcrypt registered [ 3136.313301] Key type .llcrypt registered [ 3136.438325] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3137.392354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3137.578828] alg: No test for adler32 (adler32-zlib) [ 3138.644468] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 3138.962636] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3140.648185] Key type lgssc registered [ 3141.885491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3147.906824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3147.911790] LustreError: 103712:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3147.937855] LustreError: 103719:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3147.942117] LustreError: 103712:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3147.947595] LustreError: 103712:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3147.956842] LustreError: 103712:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3147.965571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.373721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.726927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.106136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3163.903988] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3171.288858] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing lsmod [ 3174.598801] LustreError: 104939:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3174.601529] LustreError: 104938:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3174.606121] LustreError: 104939:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3174.608856] LustreError: 104939:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3174.613918] LustreError: 104939:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3174.616196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.919136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.465787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3178.480921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.513794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.004059] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.007864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.027891] Lustre: Skipped 1 previous similar message [ 3180.052746] Lustre: Skipped 2 previous similar messages [ 3183.012533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3183.028335] Lustre: Skipped 2 previous similar messages [ 3189.225273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.229799] Lustre: Skipped 5 previous similar messages [ 3192.297442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3192.520660] Lustre: server umount lustre-MDT0000 complete [ 3194.340291] LustreError: 103751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.375134] LustreError: 103751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3196.169985] LustreError: 105123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300893 with bad export cookie 5837329951789266495 [ 3196.193101] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.351559] LustreError: 104189:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b6c8ae9 x1876217913433344/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3196.557820] Lustre: server umount lustre-MDT0001 complete [ 3210.701782] Lustre: server umount lustre-OST0000 complete [ 3219.469384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.157572] LustreError: 106004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.283495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.582580] LustreError: 106005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.592875] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3230.702672] LustreError: 106004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.821530] LustreError: 106005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.076566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.292390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3243.941403] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3253.128225] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3259.453993] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3266.449474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.900955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.560412] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3281.253642] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.431605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3298.786677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.792971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.813294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.833494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.843471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.052132] Lustre: server umount lustre-OST0000 complete [ 3310.050474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3310.056579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.079631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.088355] Lustre: Skipped 2 previous similar messages [ 3315.660804] Lustre: server umount lustre-MDT0000 complete [ 3316.206947] LustreError: 106005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.225783] LustreError: 106005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3319.297157] LustreError: 105984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301016 with bad export cookie 5837329951789267195 [ 3319.305456] LustreError: 105984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3319.317035] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.489534] Lustre: server umount lustre-MDT0001 complete [ 3332.951744] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3335.775800] Key type lgssc unregistered [ 3336.089685] LNet: 109558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3336.101440] LNetError: 109558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3336.131027] LNet: Removed LNI 192.168.202.155@tcp [ 3337.374846] Key type .llcrypt unregistered [ 3337.380898] Key type ._llcrypt unregistered [ 3364.128466] Key type ._llcrypt registered [ 3364.130255] Key type .llcrypt registered [ 3364.245578] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3365.656345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.723598] alg: No test for adler32 (adler32-zlib) [ 3366.812649] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 3367.152560] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3368.952327] Key type lgssc registered [ 3370.221681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3377.940502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3377.962522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.460254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.181735] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3389.925545] LustreError: 110973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.940534] LustreError: 110973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3393.571182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.030129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3398.936316] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3408.302389] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3416.766435] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3426.090515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.456766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.315354] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3442.796608] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3450.137653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3459.307611] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3464.763229] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3469.794119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.806595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.817058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3470.820279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3470.823091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3470.834674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.428132] Lustre: server umount lustre-OST0000 complete [ 3481.060219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.061431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3481.091174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3481.100315] Lustre: Skipped 2 previous similar messages [ 3485.836678] Lustre: server umount lustre-MDT0000 complete [ 3486.177295] LustreError: 110974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.057539] LustreError: 113531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301187 with bad export cookie 1825396593388043243 [ 3490.060818] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.065364] LustreError: 113531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3490.515855] Lustre: server umount lustre-MDT0001 complete [ 3500.911787] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3503.388452] Key type lgssc unregistered [ 3503.682533] LNet: 114636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3503.687342] LNetError: 114636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3503.697555] LNet: Removed LNI 192.168.202.155@tcp [ 3504.543226] Key type .llcrypt unregistered [ 3504.548957] Key type ._llcrypt unregistered [ 3524.329828] Key type ._llcrypt registered [ 3524.331537] Key type .llcrypt registered [ 3524.415224] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3525.527086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3525.625098] alg: No test for adler32 (adler32-zlib) [ 3526.722665] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 3526.962283] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3528.648506] Key type lgssc registered [ 3530.135378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3538.373773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3538.412897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.102789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.129307] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3550.695368] LustreError: 115853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.717294] LustreError: 115853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3554.609982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.888809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.535888] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3567.848350] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3573.970995] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3581.060844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.526788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.663783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3587.944940] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3595.419355] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3609.065770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3609.073527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.098196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3609.576693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3609.596439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.618919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.703068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.709390] Lustre: Skipped 1 previous similar message [ 3619.811547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.821834] Lustre: Skipped 1 previous similar message [ 3621.346033] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3621.532756] Lustre: server umount lustre-OST0000 complete [ 3626.465666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3626.474078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.497660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.069960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.105436] Lustre: server umount lustre-MDT0000 complete [ 3635.169688] LustreError: 116593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.188306] LustreError: 116593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3636.570527] LustreError: 115835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301333 with bad export cookie 4822194873001305302 [ 3636.571393] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.575930] LustreError: 115835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.793198] Lustre: server umount lustre-MDT0001 complete [ 3650.117107] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3653.993807] Key type lgssc unregistered [ 3654.267275] LNet: 119392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3654.274687] LNetError: 119392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3654.294123] LNet: Removed LNI 192.168.202.155@tcp [ 3655.406083] Key type .llcrypt unregistered [ 3655.408364] Key type ._llcrypt unregistered [ 3682.076406] Key type ._llcrypt registered [ 3682.081403] Key type .llcrypt registered [ 3682.217578] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3683.570399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3683.600164] alg: No test for adler32 (adler32-zlib) [ 3684.686616] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 3684.948464] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3686.688199] Key type lgssc registered [ 3687.920687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3696.118820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3696.153972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.634499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.596820] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3707.878207] LustreError: 120806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.906945] LustreError: 120806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3711.976737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.480236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3717.907363] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3727.194133] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3733.568041] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3740.303089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.628862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.737040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3748.052367] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3756.101170] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3762.507466] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3767.552914] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3767.557211] Lustre: Skipped 2 previous similar messages [ 3770.983577] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3771.049589] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3772.779874] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3772.786154] Lustre: Skipped 2 previous similar messages [ 3781.274409] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3781.276545] Lustre: Skipped 1 previous similar message [ 3787.340779] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3787.349160] Lustre: Skipped 2 previous similar messages [ 3792.941697] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.55@tcp 192.168.0.[2,10]@tcp [ 3799.750322] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.55/24@tcp in log lustre-MDT0000 [ 3799.764380] Lustre: Skipped 2 previous similar messages [ 3802.808639] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.55/24@tcp [ 3802.827575] Lustre: Skipped 1 previous similar message [ 3810.792796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3810.795305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.804162] LustreError: Skipped 2 previous similar messages [ 3810.830865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.349020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.350114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.373516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.378927] LustreError: Skipped 3 previous similar messages [ 3815.586095] Lustre: server umount lustre-OST0000 complete [ 3820.007927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3820.022495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.047745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.059536] Lustre: Skipped 1 previous similar message [ 3823.596963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.600600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.650188] Lustre: Skipped 1 previous similar message [ 3826.112911] Lustre: server umount lustre-MDT0000 complete [ 3828.706410] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.735373] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3830.018733] LustreError: 120786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301527 with bad export cookie 79204000392318976 [ 3830.027972] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3830.028715] LustreError: 120786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3830.471420] Lustre: server umount lustre-MDT0001 complete [ 3842.509533] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3845.508658] Key type lgssc unregistered [ 3845.816672] LNet: 125798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3845.827948] LNetError: 125798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3845.847147] LNet: Removed LNI 192.168.202.155@tcp [ 3846.833131] Key type .llcrypt unregistered [ 3846.835426] Key type ._llcrypt unregistered [ 3880.159404] Key type ._llcrypt registered [ 3880.164927] Key type .llcrypt registered [ 3880.282485] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3881.575753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3881.615652] alg: No test for adler32 (adler32-zlib) [ 3882.749342] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 3882.947775] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3884.644116] Key type lgssc registered [ 3886.178759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3893.999841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3894.019500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.497337] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3895.545389] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.55/24@tcp [ 3895.579408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.270748] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3905.008181] LustreError: 127358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.029797] LustreError: 127358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3909.754921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.112161] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3910.155306] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.55/24@tcp [ 3910.219109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3914.811527] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3923.713779] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.823687] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.292107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.742709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.853528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3943.874909] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3952.241128] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.440268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.455190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.466792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.531215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.534825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.538337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.846890] Lustre: server umount lustre-OST0000 complete [ 3975.655318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.656971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.668275] Lustre: Skipped 1 previous similar message [ 3975.682860] Lustre: Skipped 2 previous similar messages [ 3980.778206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.787028] Lustre: Skipped 1 previous similar message [ 3982.013994] Lustre: server umount lustre-MDT0000 complete [ 3985.898647] LustreError: 129365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.912986] LustreError: 129365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3986.648496] LustreError: 127339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301683 with bad export cookie 3049551183618508464 [ 3986.656892] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.665543] LustreError: 127339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3987.140917] Lustre: server umount lustre-MDT0001 complete [ 4003.178285] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 4007.069565] Key type lgssc unregistered [ 4007.439557] LNet: 130951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4007.450705] LNetError: 130951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4007.471552] LNet: Removed LNI 192.168.202.155@tcp [ 4008.715088] Key type .llcrypt unregistered [ 4008.717985] Key type ._llcrypt unregistered [ 4044.374815] Key type ._llcrypt registered [ 4044.376441] Key type .llcrypt registered [ 4044.459493] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 4061.707275] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4062.837550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4062.965660] alg: No test for adler32 (adler32-zlib) [ 4064.128717] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 4064.349171] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 4066.080214] Key type lgssc registered [ 4067.710728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4112.902639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4112.918213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.236378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4114.305495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4114.417226] Lustre: lustre-MDT0000: new disk, initializing [ 4114.532065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.595326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.909500] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4135.568166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.722831] Lustre: 134351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4135.779538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4135.786851] Lustre: Skipped 1 previous similar message [ 4135.862828] Lustre: lustre-MDT0001: new disk, initializing [ 4135.954924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.992733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4136.003186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4142.317085] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4154.993364] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4160.938547] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.278352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.616639] Lustre: lustre-OST0000: new disk, initializing [ 4168.621670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4168.630444] Lustre: 136321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.724618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.934582] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4176.011479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4176.021372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4176.198235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4184.896422] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4189.518628] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4189.780131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4194.423167] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4194.629341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4196.833783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.845661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.859085] Lustre: Skipped 1 previous similar message [ 4196.866972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.877254] Lustre: Skipped 1 previous similar message [ 4201.957206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4201.963047] Lustre: Skipped 1 previous similar message [ 4202.756764] Lustre: server umount lustre-OST0000 complete [ 4207.074744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.085850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.093837] Lustre: Skipped 1 previous similar message [ 4212.200067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.213753] Lustre: Skipped 2 previous similar messages [ 4212.655809] Lustre: server umount lustre-MDT0000 complete [ 4216.664425] LustreError: 134344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301913 with bad export cookie 9359770838566479098 [ 4216.666143] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.671192] LustreError: 134344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4217.116403] Lustre: server umount lustre-MDT0001 complete [ 4226.382778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.726689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.801558] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4232.161263] LustreError: 138539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.178619] LustreError: 138539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4236.261397] LustreError: 138540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.207342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.378379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4241.555311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.093876] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4253.958948] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4260.459691] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.336670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.932817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.475733] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4289.277723] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4294.037507] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4294.287589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4300.096631] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4300.370670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4310.088333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4310.310777] Lustre: lustre-OST0001: new disk, initializing [ 4310.314891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4310.320832] Lustre: 141909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4310.419333] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4317.221085] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4317.279397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4317.300764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4317.463364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4326.401219] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4326.562633] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4331.620685] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4331.824411] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4338.101426] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927f46ffe800.ost_server_uuid 50 [ 4339.755405] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff927f46ffe800.ost_server_uuid in IDLE state after 0 sec [ 4353.512922] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.523870] LustreError: Skipped 1 previous similar message [ 4353.537099] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.554321] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4353.566874] Lustre: Skipped 1 previous similar message [ 4354.896668] Lustre: server umount lustre-OST0001 complete [ 4358.629633] LustreError: 140360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.642280] LustreError: 140360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4363.235494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.241954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.247027] LustreError: Skipped 2 previous similar messages [ 4363.264380] Lustre: Skipped 2 previous similar messages [ 4363.748193] LustreError: 140361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.769770] LustreError: 140361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4365.685938] Lustre: server umount lustre-OST0000 complete [ 4374.004733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.023162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4374.032226] Lustre: Skipped 1 previous similar message [ 4374.053394] Lustre: Skipped 3 previous similar messages [ 4385.253099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4385.434985] Lustre: server umount lustre-MDT0000 complete [ 4388.325003] LustreError: 138535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.345798] LustreError: 138535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4390.195946] LustreError: 138520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302087 with bad export cookie 9359770838566479973 [ 4390.210699] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.569657] Lustre: server umount lustre-MDT0001 complete [ 4414.470088] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 4417.260228] Key type lgssc unregistered [ 4417.648710] LNet: 144913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4417.659765] LNetError: 144913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4417.686358] LNet: Removed LNI 192.168.202.155@tcp [ 4418.669894] Key type .llcrypt unregistered [ 4418.675115] Key type ._llcrypt unregistered [ 4463.122232] Key type ._llcrypt registered [ 4463.127351] Key type .llcrypt registered [ 4463.217400] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 4481.140194] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4482.643869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4482.676236] alg: No test for adler32 (adler32-zlib) [ 4483.861191] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 4484.221772] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 4486.048303] Key type lgssc registered [ 4487.634888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.842058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4524.877216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.211367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4526.269862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4526.388906] Lustre: lustre-MDT0000: new disk, initializing [ 4526.561357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.578739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4531.641572] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4542.694757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.882884] Lustre: 149068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4542.950248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4542.953600] Lustre: Skipped 1 previous similar message [ 4543.025172] Lustre: lustre-MDT0001: new disk, initializing [ 4543.122015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4543.219299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4543.236096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4547.755461] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4558.254500] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4564.686568] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4572.618441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.955499] Lustre: lustre-OST0000: new disk, initializing [ 4572.959102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4572.965914] Lustre: 151035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4573.038169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.949303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4578.958580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4579.187831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4581.907315] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4593.011744] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4598.977774] Lustre: Failing over lustre-OST0000 [ 4599.241224] Lustre: server umount lustre-OST0000 complete [ 4615.137651] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302296/real 1789302296] req@00000000add04221 x1876219323780480/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789302312 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4615.140158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.209205] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4620.812203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4621.277817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.308896] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4621.381283] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302302/real 1789302302] req@0000000074458f72 x1876219323781504/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789302318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.424923] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4625.418483] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4626.400121] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302307/real 1789302307] req@00000000406c529b x1876219323782528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789302323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4626.462954] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4627.641509] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4627.645400] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4630.454128] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4631.778801] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302312/real 1789302312] req@0000000003ca8322 x1876219323783552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789302328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4631.814141] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4635.787582] Lustre: Failing over lustre-MDT0000 [ 4636.143139] Lustre: server umount lustre-MDT0000 complete [ 4640.741341] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.741788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.778646] Lustre: Skipped 3 previous similar messages [ 4642.921918] LustreError: 149073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.944352] LustreError: 149073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4645.858388] LustreError: 151957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.877660] LustreError: 151957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4648.043417] LustreError: 149073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4650.983235] LustreError: 150166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.001210] LustreError: 150166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4656.061941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.103919] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302337/real 1789302337] req@000000000841a9f4 x1876219323795072/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789302353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.113110] LustreError: 149078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.121302] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4656.181528] LustreError: 149078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4664.438522] LustreError: 149072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.458753] LustreError: 149072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4666.353336] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf7dcf93c518b5ab4 [ 4666.372739] Lustre: MGC192.168.202.155@tcp: Connection restored to 0@lo (at 0@lo) [ 4666.386522] Lustre: Skipped 1 previous similar message [ 4666.605135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.656406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4669.548835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4671.327827] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4672.006057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4672.014520] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4672.088188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4679.137403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4679.143543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.161061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4682.213396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4682.216661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.196437] Lustre: server umount lustre-OST0000 complete [ 4692.449386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4692.449987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.458854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.458862] Lustre: Skipped 1 previous similar message [ 4692.477260] Lustre: Skipped 1 previous similar message [ 4694.644234] Lustre: server umount lustre-MDT0000 complete [ 4697.572092] LustreError: 151957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.595114] LustreError: 151957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4698.410699] LustreError: 149060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302395 with bad export cookie 17860424259705985716 [ 4698.411268] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4698.427959] LustreError: 149060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4699.048126] Lustre: server umount lustre-MDT0001 complete [ 4711.707495] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 4714.559601] Key type lgssc unregistered [ 4714.811264] LNet: 154695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4714.825052] LNetError: 154695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4714.844909] LNet: Removed LNI 192.168.202.155@tcp [ 4715.811755] Key type .llcrypt unregistered [ 4715.813505] Key type ._llcrypt unregistered [ 4754.389226] Key type ._llcrypt registered [ 4754.390923] Key type .llcrypt registered [ 4754.475449] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4755.815143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.849368] alg: No test for adler32 (adler32-zlib) [ 4757.123156] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 4757.463272] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 4759.176558] Key type lgssc registered [ 4760.591801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4769.921161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4769.940055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.325994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.129512] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4781.542675] LustreError: 156213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4781.583045] LustreError: 156213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4785.620374] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4786.672570] LustreError: 156214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.687737] LustreError: 156242:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4789.697697] LustreError: 156242:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4789.703620] LustreError: 156242:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4790.112357] Lustre: server umount lustre-MDT0000 complete [ 4806.043247] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 4817.460713] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4855.974195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.391248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4856.421276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4856.569069] Lustre: lustre-MDT0000: new disk, initializing [ 4856.697355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.733691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4863.472453] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4875.208226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.330317] Lustre: 159547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4875.371343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4875.374539] Lustre: Skipped 1 previous similar message [ 4875.462928] Lustre: lustre-MDT0001: new disk, initializing [ 4875.571867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4875.624821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4875.642171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4881.131580] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4892.001743] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4898.924370] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4906.703950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4907.129929] Lustre: lustre-OST0000: new disk, initializing [ 4907.133204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4907.138615] Lustre: 161516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4907.240265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.373522] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 4915.818740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4915.837668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4915.951180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4922.986387] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4927.345803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4929.018312] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:30:25 (1789302625) [ 5180.385746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.387886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5180.401224] Lustre: Skipped 1 previous similar message [ 5180.422685] Lustre: Skipped 2 previous similar messages [ 5185.506212] LustreError: 159552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.531781] LustreError: 159552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5185.560790] Lustre: server umount lustre-MDT0000 complete [ 5189.191097] LustreError: 159539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302886 with bad export cookie 2059853098833193677 [ 5189.195018] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.198238] LustreError: 159539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5190.627435] LustreError: 159553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.627496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.677958] Lustre: Skipped 1 previous similar message [ 5190.685709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5194.656721] LustreError: 163290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.657467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5195.481416] Lustre: server umount lustre-MDT0001 complete [ 5205.167352] Lustre: server umount lustre-OST0000 complete [ 5214.421756] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5223.968283] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5262.738655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.990981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5263.016324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5263.078885] Lustre: lustre-MDT0000: new disk, initializing [ 5263.134041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.157486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5267.720335] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5277.950202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.058510] Lustre: 166193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5278.102818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5278.106362] Lustre: Skipped 1 previous similar message [ 5278.180826] Lustre: lustre-MDT0001: new disk, initializing [ 5278.301692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.350600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5278.365590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5282.649184] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5292.483242] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5298.650343] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5305.862588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.265347] Lustre: lustre-OST0000: new disk, initializing [ 5306.268527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5306.279258] Lustre: 168160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5306.355403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.630671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5312.647715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5312.763528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5313.920934] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5326.418215] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5332.684352] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5332.970830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5338.987915] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5339.271683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5343.718814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.727367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.739714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.758406] Lustre: Skipped 1 previous similar message [ 5347.639642] Lustre: server umount lustre-OST0000 complete [ 5353.955440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.972827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.985480] Lustre: Skipped 2 previous similar messages [ 5354.001681] Lustre: Skipped 1 previous similar message [ 5358.892699] Lustre: server umount lustre-MDT0000 complete [ 5359.081033] LustreError: 166203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5362.763141] LustreError: 166186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303059 with bad export cookie 2059853098833352514 [ 5362.773898] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.776269] LustreError: 166186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5363.152930] Lustre: server umount lustre-MDT0001 complete [ 5373.769535] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:37:50 (1789303070) [ 5391.053343] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5400.306547] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5438.858543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.107589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5439.133354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5439.289111] Lustre: lustre-MDT0000: new disk, initializing [ 5439.419872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5439.453081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.822766] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5454.883654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.170589] Lustre: 172916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5455.585634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5461.666795] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5473.183903] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.139471] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5487.703931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5488.254190] Lustre: lustre-OST0000: new disk, initializing [ 5488.260773] Lustre: Skipped 1 previous similar message [ 5488.269628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5488.273242] Lustre: Skipped 2 previous similar messages [ 5488.277348] Lustre: 174894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5493.890826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5493.895967] Lustre: Skipped 1 previous similar message [ 5493.900799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5494.045192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5495.756879] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5506.756870] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5514.739508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5514.744382] LustreError: Skipped 1 previous similar message [ 5514.747551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.758411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5520.544746] Lustre: server umount lustre-OST0000 complete [ 5524.962811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.972020] Lustre: Skipped 2 previous similar messages [ 5530.788201] Lustre: server umount lustre-MDT0000 complete [ 5534.440807] LustreError: 172908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303231 with bad export cookie 2059853098833353396 [ 5534.454676] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5534.911772] Lustre: server umount lustre-MDT0001 complete [ 5543.516304] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:40:40 (1789303240) [ 5559.941444] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5569.933429] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5615.890727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.283410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.356553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5616.493469] Lustre: lustre-MDT0000: new disk, initializing [ 5616.611037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.620803] Lustre: Skipped 2 previous similar messages [ 5616.644578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5621.664345] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5632.171145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.334439] Lustre: 179219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5632.571902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5637.220809] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5645.567956] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5651.717880] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5658.022597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5658.409121] Lustre: 181191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5664.899836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5664.984606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5665.338660] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5679.704562] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5690.853124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5690.856825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.858308] LustreError: Skipped 2 previous similar messages [ 5690.867057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5690.870127] Lustre: Skipped 8 previous similar messages [ 5694.081241] Lustre: server umount lustre-OST0000 complete [ 5699.046219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5704.998257] Lustre: server umount lustre-MDT0000 complete [ 5706.209157] LustreError: 179228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.217444] LustreError: 179228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5708.518964] LustreError: 179209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303405 with bad export cookie 2059853098833354481 [ 5708.525587] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5708.530264] LustreError: 179209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5708.792222] Lustre: server umount lustre-MDT0001 complete [ 5719.954520] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 5722.775052] Key type lgssc unregistered [ 5723.146051] LNet: 183049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5723.156053] LNetError: 183049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5723.168245] LNet: Removed LNI 192.168.202.155@tcp [ 5724.184304] Key type .llcrypt unregistered [ 5724.195422] Key type ._llcrypt unregistered [ 5747.681887] Key type ._llcrypt registered [ 5747.684273] Key type .llcrypt registered [ 5747.775390] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5748.650955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.738124] alg: No test for adler32 (adler32-zlib) [ 5749.811127] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 5750.045104] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 5751.762782] Key type lgssc registered [ 5752.923443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5760.155906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5760.169292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.540135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.838936] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5771.750569] LustreError: 184460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.783806] LustreError: 184460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5774.357139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.733311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5779.779128] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5788.987449] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5796.068273] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5805.089150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.557077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.605306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5813.499470] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5823.917357] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5842.401858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5842.415614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.437522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5845.488308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5845.497755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5845.520853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.460822] Lustre: server umount lustre-OST0000 complete [ 5855.722587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.733111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.739391] Lustre: Skipped 1 previous similar message [ 5855.758223] Lustre: Skipped 2 previous similar messages [ 5859.776727] Lustre: server umount lustre-MDT0000 complete [ 5860.836731] LustreError: 184460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.881533] LustreError: 184460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5864.781216] LustreError: 184442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303561 with bad export cookie 13723798504651739247 [ 5864.793149] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5864.793330] LustreError: 184442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5865.218779] Lustre: server umount lustre-MDT0001 complete [ 5878.258663] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 5881.679296] Key type lgssc unregistered [ 5881.996061] LNet: 188016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.001955] LNetError: 188016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5882.017398] LNet: Removed LNI 192.168.202.155@tcp [ 5883.238872] Key type .llcrypt unregistered [ 5883.243706] Key type ._llcrypt unregistered [ 5908.865600] Key type ._llcrypt registered [ 5908.867352] Key type .llcrypt registered [ 5909.053495] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5910.631746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.852143] alg: No test for adler32 (adler32-zlib) [ 5912.105946] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 5912.538422] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 5914.321084] Key type lgssc registered [ 5916.225053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5926.573387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5926.608408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.177675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.452533] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5938.659933] LustreError: 189433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.673335] LustreError: 189433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5943.030942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.468422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5949.067563] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5956.316246] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5962.162750] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5969.858077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5970.271717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.367563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5976.218192] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5983.723201] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.685104] Lustre: lustre-OST0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 5999.080241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.083323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5999.133658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5999.156407] Lustre: Skipped 1 previous similar message [ 6001.776283] Lustre: lustre-OST0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 6004.201531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.209653] Lustre: Skipped 1 previous similar message [ 6008.288162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6008.511805] Lustre: server umount lustre-OST0000 complete [ 6016.757776] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff927f60347000.ost_server_uuid 50 [ 6018.499430] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6034.914301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.921469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.929328] Lustre: Skipped 2 previous similar messages [ 6034.963625] Lustre: Skipped 2 previous similar messages [ 6038.622773] Lustre: server umount lustre-MDT0000 complete [ 6040.038369] LustreError: 189427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.083112] LustreError: 189427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6042.483709] LustreError: 189414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303739 with bad export cookie 5812181756243994157 [ 6042.492117] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.496348] LustreError: 189414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6042.865312] Lustre: server umount lustre-MDT0001 complete [ 6051.203662] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:49:07 (1789303747) [ 6058.291395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.828768] LustreError: 193359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.899392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.587633] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6064.100561] LustreError: 193360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.195454] LustreError: 193359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.017410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.444270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6077.671371] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6085.124295] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.778937] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.159130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.569517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.614355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6108.415522] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6117.623381] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6124.848830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.058705] Lustre: lustre-OST0001: new disk, initializing [ 6125.065525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6125.071253] Lustre: 196199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6125.152084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6131.885510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6131.899461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6132.058072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6133.602196] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6146.543511] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6149.603016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6149.610612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.633194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6152.682877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.310842] Lustre: server umount lustre-OST0000 complete [ 6157.424484] LustreError: 196386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.564093] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6162.780236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6167.420516] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6167.600693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6177.916192] LustreError: 196386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.936805] LustreError: 196386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6183.404801] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6183.412820] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.425604] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6183.432117] Lustre: Skipped 4 previous similar messages [ 6187.705808] Lustre: server umount lustre-OST0001 complete [ 6193.634351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.649555] Lustre: Skipped 2 previous similar messages [ 6199.014453] Lustre: server umount lustre-MDT0000 complete [ 6203.258827] LustreError: 193341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303900 with bad export cookie 5812181756243995228 [ 6203.266085] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.282415] LustreError: 193341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6203.794906] Lustre: server umount lustre-MDT0001 complete [ 6229.638662] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:52:06 (1789303926) [ 6235.992184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.216686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6236.243852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6236.448158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.160209] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6249.445080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.517879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6249.553761] Lustre: 199643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6254.791526] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6263.282907] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.023896] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.079437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.384653] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6285.476090] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6285.478588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6293.717944] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.952190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6300.998693] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6301.281157] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6301.289700] Lustre: Skipped 2 previous similar messages [ 6306.963805] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6307.683334] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6315.497778] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.121827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6321.133237] LustreError: Skipped 1 previous similar message [ 6321.139763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.174629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6321.182655] Lustre: Skipped 4 previous similar messages [ 6324.825950] Lustre: server umount lustre-OST0000 complete [ 6326.907573] LustreError: 201496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.936333] LustreError: 201496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6342.631756] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6342.653934] LustreError: Skipped 1 previous similar message [ 6347.071685] Lustre: server umount lustre-OST0001 complete [ 6353.380239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.395774] Lustre: Skipped 4 previous similar messages [ 6359.136716] Lustre: server umount lustre-MDT0000 complete [ 6365.018174] LustreError: 199635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304062 with bad export cookie 5812181756243996635 [ 6365.021559] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.041994] LustreError: 199635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6365.601898] Lustre: server umount lustre-MDT0001 complete [ 6397.476879] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:54:54 (1789304094) [ 6416.757772] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 6427.990401] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6473.376719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.657977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6473.690434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6473.792893] Lustre: lustre-MDT0000: new disk, initializing [ 6473.944084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.960292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6477.707875] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6487.410692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.572125] Lustre: 207740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6487.608459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6487.611079] Lustre: Skipped 1 previous similar message [ 6487.656417] Lustre: lustre-MDT0001: new disk, initializing [ 6487.758151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6487.767805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6493.072037] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6503.091159] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6510.596363] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6518.740639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.339540] Lustre: lustre-OST0000: new disk, initializing [ 6519.343458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6519.348898] Lustre: 209714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6525.501138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6525.509719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6525.668975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6526.580948] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6536.995680] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.538671] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6542.764994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6547.453195] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6547.664145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6551.521975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6551.525408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6551.532784] LustreError: Skipped 2 previous similar messages [ 6551.536574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6551.554515] Lustre: Skipped 1 previous similar message [ 6551.586949] Lustre: Skipped 12 previous similar messages [ 6555.707829] Lustre: server umount lustre-OST0000 complete [ 6565.784238] Lustre: server umount lustre-MDT0000 complete [ 6566.880957] LustreError: 207753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.893588] LustreError: 207753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6569.534510] LustreError: 207733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304266 with bad export cookie 5812181756243998420 [ 6569.538312] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.545312] LustreError: 207733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6569.821939] Lustre: server umount lustre-MDT0001 complete [ 6579.019554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.505746] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6593.285957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.906972] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6606.649666] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.947647] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6622.469259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6630.467492] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6638.877334] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6643.860751] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6644.178950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6648.998414] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6649.182142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6651.364397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.376052] LustreError: Skipped 1 previous similar message [ 6657.488157] Lustre: server umount lustre-OST0000 complete [ 6664.656085] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6668.971622] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6698.208813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.787945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6703.801024] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6705.087302] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6712.611323] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6717.231804] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6718.125476] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6718.146202] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6722.997648] Lustre: server umount lustre-OST0000 complete [ 6726.626025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6726.632661] LustreError: Skipped 1 previous similar message [ 6726.637687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.648609] Lustre: Skipped 4 previous similar messages [ 6726.654609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.659200] Lustre: Skipped 6 previous similar messages [ 6737.197157] Lustre: server umount lustre-MDT0000 complete [ 6738.912207] Lustre: 188742:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304420/real 1789304420] req@00000000d9321778 x1876220821373312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789304436 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6738.919260] LustreError: 211931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.940511] Lustre: 188742:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6738.949152] LustreError: 211931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6741.548642] LustreError: 211912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304438 with bad export cookie 5812181756243999302 [ 6741.554386] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6741.940920] Lustre: server umount lustre-MDT0001 complete [ 6751.710482] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:00:48 (1789304448) [ 6758.576649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.264372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.274149] Lustre: Skipped 6 previous similar messages [ 6764.404370] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6771.881670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.728975] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6783.661228] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6790.029519] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.520823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.440438] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6810.832866] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6816.179249] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6816.419331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6820.487748] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6820.729061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6827.855956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.070261] Lustre: lustre-OST0001: new disk, initializing [ 6828.073399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6828.078693] Lustre: 221274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6834.282062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6834.289949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6834.450453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6834.899824] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6844.977711] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.407136] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6850.657315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6855.779484] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6856.052201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6860.257202] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.273045] LustreError: Skipped 1 previous similar message [ 6864.161308] Lustre: server umount lustre-OST0001 complete [ 6870.996961] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6871.169209] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6876.233660] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6876.518771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6907.165779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.000721] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6913.023860] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6913.024952] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6915.261232] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6923.662881] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6925.463663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6942.176675] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.338338] Lustre: server umount lustre-OST0001 complete [ 6965.798996] Lustre: server umount lustre-OST0000 complete [ 6976.537503] Lustre: server umount lustre-MDT0000 complete [ 6980.379689] LustreError: 217876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304677 with bad export cookie 5812181756244000268 [ 6980.385354] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6980.395578] LustreError: 217876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6986.833856] Lustre: server umount lustre-MDT0001 complete [ 7010.398706] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:05:07 (1789304707) [ 7018.551886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7018.768216] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7018.794575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7024.187162] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7033.659558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.733950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7033.772121] Lustre: 226520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7033.853606] LustreError: 226532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.876586] LustreError: 226532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7040.002103] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7048.790110] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7054.998822] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7061.946608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.116971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7069.029219] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7077.002528] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7088.393961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.564814] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7095.933831] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7104.221587] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7106.197132] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7111.303621] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7111.499539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7117.090527] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7117.454223] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7123.459785] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927f448fb800.ost_server_uuid 50 [ 7125.361898] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff927f448fb800.ost_server_uuid in IDLE state after 0 sec [ 7126.556245] Lustre: Permanently deactivating lustre-OST0001 [ 7126.580414] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7130.793976] Lustre: Permanently reactivating lustre-OST0001 [ 7130.811767] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7130.818842] Lustre: Skipped 2 previous similar messages [ 7135.236775] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.241872] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7135.242827] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7135.242833] LustreError: Skipped 5 previous similar messages [ 7135.268172] Lustre: Skipped 11 previous similar messages [ 7135.301841] Lustre: Skipped 19 previous similar messages [ 7140.584299] Lustre: server umount lustre-OST0001 complete [ 7150.962699] Lustre: server umount lustre-OST0000 complete [ 7161.237288] Lustre: server umount lustre-MDT0000 complete [ 7164.834207] LustreError: 226512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304861 with bad export cookie 5812181756244001556 [ 7164.840315] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7164.846239] LustreError: 226512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7165.248653] Lustre: server umount lustre-MDT0001 complete [ 7190.702327] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:08:07 (1789304887) [ 7199.936529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.296901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7200.341322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7200.348789] Lustre: Skipped 2 previous similar messages [ 7205.643609] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7214.549538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.653249] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7214.693778] Lustre: 233217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7219.764786] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7226.737648] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7232.326544] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7239.296478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.539911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7239.571540] Lustre: Permanently deactivating lustre-OST0000 [ 7239.585906] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7246.126127] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7246.397707] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7254.173528] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7260.667145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7260.764637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7267.040833] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7270.123143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7275.946138] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7277.819321] Lustre: Permanently reactivating lustre-OST0000 [ 7277.846560] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7277.858614] Lustre: Skipped 2 previous similar messages [ 7278.477940] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7278.478541] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000ed9c2a52, cur 1789304975 deadline 1789304973 last 1789304943 [ 7278.491298] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7278.511065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7278.515677] Lustre: Skipped 1 previous similar message [ 7278.600594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7301.748377] Lustre: server umount lustre-OST0001 complete [ 7311.886477] Lustre: server umount lustre-OST0000 complete [ 7321.828153] Lustre: server umount lustre-MDT0000 complete [ 7325.545910] LustreError: 233211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305022 with bad export cookie 5812181756244003215 [ 7325.547073] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7325.561288] LustreError: 233211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7326.221527] Lustre: server umount lustre-MDT0001 complete [ 7338.486449] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 7341.214775] Key type lgssc unregistered [ 7341.530532] LNet: 238104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7341.539373] LNetError: 238104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7341.557420] LNet: Removed LNI 192.168.202.155@tcp [ 7342.434674] Key type .llcrypt unregistered [ 7342.436555] Key type ._llcrypt unregistered [ 7366.214794] Key type ._llcrypt registered [ 7366.220385] Key type .llcrypt registered [ 7366.331657] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7367.333732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.678886] alg: No test for adler32 (adler32-zlib) [ 7368.754342] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 7369.046351] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 7370.736804] Key type lgssc registered [ 7371.938651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7381.577765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7381.600814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.206565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.724448] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7393.763529] LustreError: 239566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.796000] LustreError: 239566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7396.583174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.750825] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7396.782816] Lustre: 239554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7396.792868] Lustre: Permanently deactivating lustre-MDT0001 [ 7396.811472] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7397.124591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7402.659807] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7405.601345] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7409.912753] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.851504] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7421.414819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7421.760103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7422.770980] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.791120] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7426.254633] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7428.069260] LustreError: 241408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.123521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7433.156990] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7434.214903] LustreError: 241912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.240063] LustreError: 241912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7439.338218] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.352637] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7439.930591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7440.087321] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.507188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7445.508398] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7445.852954] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7454.194973] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.391647] Lustre: Permanently reactivating lustre-MDT0001 [ 7458.401196] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7458.405307] Lustre: Skipped 2 previous similar messages [ 7463.579133] Lustre: lustre-MDT0001: Client 451f76f1-a582-4c50-a8b2-6762a9224e41 (at 192.168.202.55@tcp) reconnecting [ 7468.619903] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.627903] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7468.631967] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7468.639213] LustreError: 243277:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7468.641744] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7468.649220] LustreError: 243277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7468.650495] LustreError: 243277:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.155@tcp: cfg command failed: rc = -108 [ 7468.681564] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7468.681564] [ 7468.686980] LustreError: 239558:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7469.841950] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7471.704501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7477.142801] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7477.358761] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7481.748439] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7481.933377] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7484.609895] Lustre: Permanently deactivating lustre-MDT0001 [ 7484.622459] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7484.625985] Lustre: Skipped 2 previous similar messages [ 7490.648363] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7499.747254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.752618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7499.755492] Lustre: Skipped 3 previous similar messages [ 7499.755654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7499.767891] Lustre: Skipped 3 previous similar messages [ 7501.067912] Lustre: server umount lustre-MDT0000 complete [ 7504.095149] LustreError: 239546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305201 with bad export cookie 3382979724099978048 [ 7504.100051] LustreError: 239546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7504.105065] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7504.323248] Lustre: server umount lustre-MDT0001 complete [ 7507.481872] Lustre: server umount lustre-OST0000 complete [ 7516.987108] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:13:34 (1789305214) [ 7525.673663] Lustre: server umount lustre-OST0001 complete [ 7531.444391] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 7538.537784] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7570.166305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.414795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7570.430623] Lustre: Skipped 2 previous similar messages [ 7570.447542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7570.509285] Lustre: lustre-MDT0000: new disk, initializing [ 7570.608116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.631420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7574.599736] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7584.402278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.488843] Lustre: 247822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7584.534168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7584.537239] Lustre: Skipped 1 previous similar message [ 7584.592350] Lustre: lustre-MDT0001: new disk, initializing [ 7584.669262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7584.699521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7584.713038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7588.891740] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7597.808821] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7603.341759] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7609.433121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.708911] Lustre: lustre-OST0000: new disk, initializing [ 7609.712261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7609.719386] Lustre: 249790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7609.794241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.436337] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7619.131524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7619.148930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7619.233287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7623.645571] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7625.396241] LustreError: 247828:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7633.158836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.338108] Lustre: lustre-OST0001: new disk, initializing [ 7633.341195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7633.349865] Lustre: 250901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7635.494033] LustreError: 247828:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7635.531636] LustreError: 250092:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7639.155940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7639.168088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7639.246867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7639.469336] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7645.616068] LustreError: 250092:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7648.753165] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.160920] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7654.883147] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7654.888135] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.897214] Lustre: Skipped 1 previous similar message [ 7654.901960] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7654.909253] Lustre: Skipped 1 previous similar message [ 7657.818920] Lustre: server umount lustre-OST0001 complete [ 7660.004938] LustreError: 249775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.017669] LustreError: 249775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7675.362153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.373732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.378819] Lustre: Skipped 1 previous similar message [ 7675.400142] Lustre: Skipped 1 previous similar message [ 7679.426390] Lustre: server umount lustre-OST0000 complete [ 7685.603848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.606038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7685.616843] Lustre: Skipped 1 previous similar message [ 7685.634398] Lustre: Skipped 1 previous similar message [ 7688.926475] Lustre: server umount lustre-MDT0000 complete [ 7690.721625] LustreError: 247833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.735115] LustreError: 247833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7692.241532] LustreError: 247813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305389 with bad export cookie 3382979724099980862 [ 7692.255793] LustreError: 247813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7692.256971] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7698.634501] Lustre: server umount lustre-MDT0001 complete [ 7708.565388] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 7711.440903] Key type lgssc unregistered [ 7711.831579] LNet: 252934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7711.836953] LNetError: 252934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7712.875944] LNet: Removed LNI 192.168.202.155@tcp [ 7713.825128] Key type .llcrypt unregistered [ 7713.827374] Key type ._llcrypt unregistered [ 7748.685916] Key type ._llcrypt registered [ 7748.688166] Key type .llcrypt registered [ 7748.783267] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7749.774560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7749.838296] alg: No test for adler32 (adler32-zlib) [ 7750.949311] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 7751.165646] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 7752.856414] Key type lgssc registered [ 7753.769909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7760.195883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.213831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.465869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7761.494765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.653556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.255399] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7771.650557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.725735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7771.752705] Lustre: 255117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7771.903948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7776.002854] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7782.602373] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7788.185828] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.654329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.815121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7795.006587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.514407] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7802.492364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7807.101992] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7858.658146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7858.669237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.701193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.180929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7859.189573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.218973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.226888] Lustre: Skipped 1 previous similar message [ 7864.398148] Lustre: server umount lustre-OST0000 complete [ 7867.702855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7877.868735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7878.103820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.304734] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7889.400221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7889.422205] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7890.785493] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7894.562737] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7894.585738] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7930.350946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7930.351491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.361895] LustreError: Skipped 1 previous similar message [ 7930.369958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.381099] Lustre: Skipped 1 previous similar message [ 7935.271824] Lustre: server umount lustre-OST0000 complete [ 7940.577806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.596130] Lustre: Skipped 1 previous similar message [ 7940.609462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.614615] Lustre: Skipped 1 previous similar message [ 7945.032885] Lustre: server umount lustre-MDT0000 complete [ 7945.698468] LustreError: 255128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.712426] LustreError: 255128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7948.231126] LustreError: 256987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305645 with bad export cookie 13944467290616484980 [ 7948.239133] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7948.245167] LustreError: 256987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7948.551821] Lustre: server umount lustre-MDT0001 complete [ 7957.304244] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 7959.733986] Key type lgssc unregistered [ 7960.008080] LNet: 260537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7960.018825] LNetError: 260537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7960.032531] LNet: Removed LNI 192.168.202.155@tcp [ 7960.585872] Key type .llcrypt unregistered [ 7960.587599] Key type ._llcrypt unregistered [ 7980.495752] Key type ._llcrypt registered [ 7980.498650] Key type .llcrypt registered [ 7980.617488] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7981.534682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.681185] alg: No test for adler32 (adler32-zlib) [ 7982.721561] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 7982.917043] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 7984.568420] Key type lgssc registered [ 7985.551801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7991.779847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7991.791745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.172183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.805905] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8003.562538] LustreError: 261930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.593624] LustreError: 261930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8003.913309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.233154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8008.133802] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8015.058827] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.169541] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.676521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.003197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.988754] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8036.642145] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8039.338391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8065.004525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.005203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.010852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.019070] LustreError: Skipped 1 previous similar message [ 8065.045681] Lustre: Skipped 1 previous similar message [ 8067.488685] Lustre: server umount lustre-OST0000 complete [ 8075.246346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.273583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8075.279783] Lustre: Skipped 2 previous similar messages [ 8077.501914] Lustre: server umount lustre-MDT0000 complete [ 8080.363435] LustreError: 261930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.396696] LustreError: 261930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8081.085119] LustreError: 261910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305778 with bad export cookie 6191497054290058965 [ 8081.107717] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8081.445146] Lustre: server umount lustre-MDT0001 complete [ 8092.675333] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8094.862871] Key type lgssc unregistered [ 8095.087140] LNet: 266232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8095.101326] LNetError: 266232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8095.122698] LNet: Removed LNI 192.168.202.155@tcp [ 8095.824456] Key type .llcrypt unregistered [ 8095.825934] Key type ._llcrypt unregistered [ 8110.211244] Key type ._llcrypt registered [ 8110.212616] Key type .llcrypt registered [ 8110.278625] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8110.986942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8111.053503] alg: No test for adler32 (adler32-zlib) [ 8112.107686] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 8112.394984] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8114.080347] Key type lgssc registered [ 8114.957719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8121.549223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8121.565583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.820755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.381841] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8132.068678] LustreError: 267449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.079430] LustreError: 267449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8133.520956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.792818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8137.150582] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8142.915625] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.126868] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8152.835582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.091717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.026726] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8164.024826] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.481918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8177.639732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.648170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8177.651117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.757801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.767188] Lustre: Skipped 2 previous similar messages [ 8183.823516] Lustre: server umount lustre-OST0000 complete [ 8187.873782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.875629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.893848] Lustre: Skipped 2 previous similar messages [ 8187.911584] Lustre: Skipped 1 previous similar message [ 8192.993916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8193.001287] Lustre: Skipped 1 previous similar message [ 8193.595652] Lustre: server umount lustre-MDT0000 complete [ 8196.775143] LustreError: 267428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305893 with bad export cookie 4399593840370362382 [ 8196.778735] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8196.783048] LustreError: 267428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8197.128964] Lustre: server umount lustre-MDT0001 complete [ 8206.389568] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8208.896286] Key type lgssc unregistered [ 8209.100867] LNet: 271123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8209.105969] LNetError: 271123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8209.125907] LNet: Removed LNI 192.168.202.155@tcp [ 8209.949064] Key type .llcrypt unregistered [ 8209.950573] Key type ._llcrypt unregistered [ 8230.671473] Key type ._llcrypt registered [ 8230.674139] Key type .llcrypt registered [ 8230.773580] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8231.800161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8231.882963] alg: No test for adler32 (adler32-zlib) [ 8232.897831] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 8233.109061] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8234.843721] Key type lgssc registered [ 8235.904780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8242.478868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8242.513409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.910751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.217084] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8253.412550] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.430615] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8253.720407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.029977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8257.987592] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8264.241810] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8268.641545] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8274.167337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8274.434595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.590959] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8283.372677] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8288.824829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8309.224229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8309.229603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.231319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8309.247249] Lustre: Skipped 1 previous similar message [ 8311.858180] Lustre: server umount lustre-OST0000 complete [ 8315.364140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8315.369929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.380148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.383695] Lustre: Skipped 1 previous similar message [ 8319.462110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8319.466839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.586507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.590731] Lustre: Skipped 2 previous similar messages [ 8329.185397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8329.367879] Lustre: server umount lustre-MDT0000 complete [ 8329.700649] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.710272] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8332.322240] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306029 with bad export cookie 1171766141864071653 [ 8332.329119] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.333182] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8332.627822] Lustre: server umount lustre-MDT0001 complete [ 8341.611600] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8343.828862] Key type lgssc unregistered [ 8344.036181] LNet: 276834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8344.040132] LNetError: 276834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8344.051661] LNet: Removed LNI 192.168.202.155@tcp [ 8344.658908] Key type .llcrypt unregistered [ 8344.661887] Key type ._llcrypt unregistered [ 8358.041661] Key type ._llcrypt registered [ 8358.043651] Key type .llcrypt registered [ 8358.135992] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8358.876432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8359.040256] alg: No test for adler32 (adler32-zlib) [ 8360.049248] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 8360.217107] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8361.848186] Key type lgssc registered [ 8362.766908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8368.856895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8368.868875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.294334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.230848] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8380.899628] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.914348] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8381.098351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.364642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8384.623283] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8389.813334] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.216860] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8399.413920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8399.608093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.151124] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8409.938299] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8414.961109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8422.374632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.385641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.396690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8425.442291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8425.445621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8425.447384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8425.465206] Lustre: Skipped 1 previous similar message [ 8428.467878] Lustre: server umount lustre-OST0000 complete [ 8432.610124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8432.619274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.628125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.688094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.689909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.630765] Lustre: server umount lustre-MDT0000 complete [ 8440.488659] LustreError: 278010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306137 with bad export cookie 1964167920311092040 [ 8440.492611] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8440.495969] LustreError: 278010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8440.764547] Lustre: server umount lustre-MDT0001 complete [ 8448.256464] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8450.217811] Key type lgssc unregistered [ 8450.410926] LNet: 281746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8450.419972] LNetError: 281746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8450.435895] LNet: Removed LNI 192.168.202.155@tcp [ 8450.963099] Key type .llcrypt unregistered [ 8450.964306] Key type ._llcrypt unregistered [ 8459.172984] Key type ._llcrypt registered [ 8459.174253] Key type .llcrypt registered [ 8459.249894] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8471.129537] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8481.182104] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8482.127817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8482.154246] alg: No test for adler32 (adler32-zlib) [ 8483.073624] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 8483.214675] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8484.832357] Key type lgssc registered [ 8485.714979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8508.344212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8508.368718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.508144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8509.528602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8509.582937] Lustre: lustre-MDT0000: new disk, initializing [ 8509.630134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.642511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8512.461746] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8520.469058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.565591] Lustre: 285270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8520.614764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8520.618104] Lustre: Skipped 1 previous similar message [ 8520.682307] Lustre: lustre-MDT0001: new disk, initializing [ 8520.756508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8520.780917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8520.796168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8524.031208] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8531.232961] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8535.416216] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8539.879582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.039923] Lustre: lustre-OST0000: new disk, initializing [ 8540.041772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8540.044545] Lustre: 287238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8540.082358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.931683] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8548.447577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8548.453913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8548.579896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8551.231743] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.530787] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8554.714727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8557.867576] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8558.047050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8564.202536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.202863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8564.214763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.229379] Lustre: Skipped 1 previous similar message [ 8565.754833] Lustre: server umount lustre-OST0000 complete [ 8569.314098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.320055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.324060] Lustre: Skipped 2 previous similar messages [ 8574.438202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.441510] Lustre: Skipped 2 previous similar messages [ 8574.943681] Lustre: server umount lustre-MDT0000 complete [ 8578.132989] LustreError: 285263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306275 with bad export cookie 5777760843289792601 [ 8578.134503] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8578.143408] LustreError: 285263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8578.406634] Lustre: server umount lustre-MDT0001 complete [ 8584.805306] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:31:22 (1789306282) [ 8589.715690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.096766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.220352] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8595.428688] LustreError: 289646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.444114] LustreError: 289646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8599.633804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.956570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8603.562149] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8609.448692] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8613.913860] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8620.180379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.540128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.312779] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8632.506836] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8647.157164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8647.167431] LustreError: Skipped 1 previous similar message [ 8647.171698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.181449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8647.183215] Lustre: Skipped 1 previous similar message [ 8647.193095] Lustre: Skipped 1 previous similar message [ 8647.680802] Lustre: server umount lustre-OST0000 complete [ 8651.240095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8651.248210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.264624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8657.374817] Lustre: server umount lustre-MDT0000 complete [ 8657.381888] LustreError: 289642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.397939] LustreError: 289642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8660.707203] LustreError: 289627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306357 with bad export cookie 5777760843289793483 [ 8660.711237] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.716380] LustreError: 289627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8660.727934] LustreError: 290384:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8660.945694] Lustre: server umount lustre-MDT0001 complete [ 8670.180552] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8672.492692] Key type lgssc unregistered [ 8672.724759] LNet: 293201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8672.728392] LNetError: 293201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8672.747613] LNet: Removed LNI 192.168.202.155@tcp [ 8673.439447] Key type .llcrypt unregistered [ 8673.441894] Key type ._llcrypt unregistered [ 8692.072765] Key type ._llcrypt registered [ 8692.074899] Key type .llcrypt registered [ 8692.144606] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8703.264976] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8704.027652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8704.220102] alg: No test for adler32 (adler32-zlib) [ 8705.198993] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 8705.379154] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8707.024215] Key type lgssc registered [ 8708.048617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8740.379082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8740.404778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.566283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8741.584795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8741.647618] Lustre: lustre-MDT0000: new disk, initializing [ 8741.698197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8741.715513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8745.029796] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8752.349451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.446525] Lustre: 297001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8752.487720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8752.490863] Lustre: Skipped 1 previous similar message [ 8752.544953] Lustre: lustre-MDT0001: new disk, initializing [ 8752.612078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8752.658703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8752.674528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8756.144833] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8762.827677] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8766.498841] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8770.967494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8771.183346] Lustre: lustre-OST2710: new disk, initializing [ 8771.186982] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8771.193073] Lustre: 298968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8771.245290] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8776.162870] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8777.289572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8777.311195] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8777.375960] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8783.135542] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8788.147667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8788.295529] Lustre: lustre-OST03e8: new disk, initializing [ 8788.298578] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8788.303709] Lustre: 300031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8788.365631] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8792.420980] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8796.724579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8796.731758] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8796.782765] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8797.432328] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8803.831536] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8803.999964] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8807.458573] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8807.645731] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8810.953870] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8811.153206] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8814.424866] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8814.626425] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8819.182523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8819.190897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.199697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.246774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.247205] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8822.257288] Lustre: Skipped 1 previous similar message [ 8822.276238] Lustre: Skipped 2 previous similar messages [ 8823.433556] Lustre: server umount lustre-MDT0000 complete [ 8825.730635] LustreError: 296991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306522 with bad export cookie 2356988997821110456 [ 8825.733816] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8825.738745] LustreError: 296991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8825.933208] Lustre: server umount lustre-MDT0001 complete [ 8839.032028] Lustre: server umount lustre-OST2710 complete [ 8851.844321] Lustre: server umount lustre-OST03e8 complete [ 8864.588561] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8871.554347] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8898.253889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.455216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8898.473447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8898.527277] Lustre: lustre-MDT0000: new disk, initializing [ 8898.617453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.630209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8901.915554] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8909.889942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.964864] Lustre: 305278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8910.004229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8910.007378] Lustre: Skipped 1 previous similar message [ 8910.060524] Lustre: lustre-MDT0001: new disk, initializing [ 8910.126517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8910.152632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8910.167356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8913.283971] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8919.975390] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8923.366883] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8927.830321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.020971] Lustre: lustre-OST0000: new disk, initializing [ 8928.024621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8928.030032] Lustre: 307243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8928.078626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.865664] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8938.031446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8938.039802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8938.156838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8938.330976] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8941.196366] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8941.378852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8944.163606] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8944.266785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8948.195938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8948.208211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.218319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8948.226111] Lustre: Skipped 2 previous similar messages [ 8951.656250] Lustre: server umount lustre-OST0000 complete [ 8956.386505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8956.390025] LustreError: Skipped 1 previous similar message [ 8956.395575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.406866] Lustre: Skipped 1 previous similar message [ 8956.412611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.416035] Lustre: Skipped 1 previous similar message [ 8960.287392] Lustre: server umount lustre-MDT0000 complete [ 8962.492868] LustreError: 305268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306659 with bad export cookie 2356988997821112129 [ 8962.495515] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.497177] LustreError: 305268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8962.646195] Lustre: server umount lustre-MDT0001 complete [ 8968.025991] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:37:45 (1789306665) [ 8969.153583] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8970.325169] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:37:47 (1789306667) [ 8972.544334] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8989.303318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.477928] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8989.490701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8989.587656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.188507] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8997.224976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.283208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8997.306533] Lustre: 311094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8997.392158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8997.394691] Lustre: Skipped 2 previous similar messages [ 9000.165772] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9018.538954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.657391] Lustre: Denying initial registration attempt from nid 192.168.202.155@tcp, specified as failover [ 9018.662670] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9018.674627] LustreError: 312546:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9018.679654] LustreError: 312546:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9018.683465] LustreError: 312546:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9018.730505] Lustre: server umount lustre-OST0000 complete [ 9018.732985] LustreError: 312546:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9023.457989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.459258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9023.464706] Lustre: Skipped 2 previous similar messages [ 9029.643596] Lustre: server umount lustre-MDT0000 complete [ 9031.778123] LustreError: 311087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306728 with bad export cookie 2356988997821113018 [ 9031.780727] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9031.786703] LustreError: 311087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9031.984318] Lustre: server umount lustre-MDT0001 complete [ 9038.181996] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9040.059615] Key type lgssc unregistered [ 9040.281861] LNet: 313665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9040.290941] LNetError: 313665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9040.304478] LNet: Removed LNI 192.168.202.155@tcp [ 9040.871643] Key type .llcrypt unregistered [ 9040.873080] Key type ._llcrypt unregistered [ 9047.498350] Key type ._llcrypt registered [ 9047.500054] Key type .llcrypt registered [ 9047.579571] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9048.197572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9048.256736] alg: No test for adler32 (adler32-zlib) [ 9049.214737] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9049.349685] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9050.960221] Key type lgssc registered [ 9051.657701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9086.493439] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9092.289302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9092.303358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.453897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9093.469536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.580495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.858215] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9103.362659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.432548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9103.461939] Lustre: 316400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9103.678288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9107.039910] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9112.675261] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9116.810628] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9134.886772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9135.032650] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9135.178972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9138.965599] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9143.759258] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9149.926719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9149.927621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.931943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9149.947560] Lustre: Skipped 1 previous similar message [ 9151.290370] Lustre: server umount lustre-OST0000 complete [ 9155.043389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9155.044558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.047081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9155.047089] Lustre: Skipped 1 previous similar message [ 9155.048140] LustreError: Skipped 1 previous similar message [ 9155.055569] Lustre: Skipped 1 previous similar message [ 9159.666867] Lustre: server umount lustre-MDT0000 complete [ 9161.789034] LustreError: 316393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306858 with bad export cookie 10794905196174757916 [ 9161.792142] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9161.801039] LustreError: 316393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9161.975667] Lustre: server umount lustre-MDT0001 complete [ 9167.012622] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9168.495104] Key type lgssc unregistered [ 9168.666752] LNet: 319970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9168.669923] LNetError: 319970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9169.708204] LNet: Removed LNI 192.168.202.155@tcp [ 9170.221103] Key type .llcrypt unregistered [ 9170.222362] Key type ._llcrypt unregistered [ 9182.534137] Key type ._llcrypt registered [ 9182.535816] Key type .llcrypt registered [ 9182.577988] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9182.897705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9182.915071] alg: No test for adler32 (adler32-zlib) [ 9183.797266] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9183.904794] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9185.496173] Key type lgssc registered [ 9185.961728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9189.747360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9189.757360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.980847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9193.220399] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9197.881568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.118276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9200.521118] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9204.557106] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9207.374426] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9212.963981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9213.149816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9216.060856] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9219.821758] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9232.494556] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 9233.781097] Lustre: server umount lustre-MDT0000 complete [ 9233.892245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.894935] LustreError: 321358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.903903] Lustre: Skipped 1 previous similar message [ 9233.908217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9233.912059] LustreError: 321358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9235.861306] LustreError: 321344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306932 with bad export cookie 11943274426452756115 [ 9235.863825] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9235.871452] LustreError: 321344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9237.609921] LustreError: 321360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.624310] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [ 9239.019050] LustreError: 321358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.019342] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.034368] Lustre: Skipped 1 previous similar message [ 9239.036815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9242.728540] LustreError: 321359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.728605] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [ 9242.736289] LustreError: 321359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9242.745895] Lustre: Skipped 1 previous similar message [ 9247.844580] LustreError: 322782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.844701] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [ 9247.863631] LustreError: 322782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9247.867199] Lustre: Skipped 1 previous similar message [ 9250.272134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9250.392071] Lustre: server umount lustre-MDT0001 complete [ 9253.337192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9257.346713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.688975] LustreError: 323193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9260.703161] LustreError: 323193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9260.749991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.790139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9262.819894] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9267.318498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.561588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9269.629516] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9273.606730] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.670971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 9279.976851] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9282.882106] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9283.892037] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9286.625550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9286.632359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.640258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9286.648633] Lustre: Skipped 1 previous similar message [ 9288.161518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9292.156208] Lustre: server umount lustre-OST0000 complete [ 9295.334474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.343726] Lustre: Skipped 2 previous similar messages [ 9300.450219] LustreError: 324978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.455792] LustreError: 324978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9300.481983] Lustre: server umount lustre-MDT0000 complete [ 9302.490942] LustreError: 321345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306999 with bad export cookie 11943274426452763248 [ 9302.493472] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9302.500980] LustreError: 321345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9302.650602] Lustre: server umount lustre-MDT0001 complete [ 9308.367197] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9309.810852] Key type lgssc unregistered [ 9309.990490] LNet: 327640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9310.000410] LNetError: 327640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9311.015765] LNet: Removed LNI 192.168.202.155@tcp [ 9311.571767] Key type .llcrypt unregistered [ 9311.575417] Key type ._llcrypt unregistered [ 9325.328297] Key type ._llcrypt registered [ 9325.330109] Key type .llcrypt registered [ 9325.386318] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9325.855595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9325.897152] alg: No test for adler32 (adler32-zlib) [ 9326.820525] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9326.937126] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9328.536975] Key type lgssc registered [ 9329.158797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9333.328661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9333.338257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.591591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.874594] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9341.475408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.675829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9344.033984] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9348.008883] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9351.263347] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9352.920623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9352.927373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9357.284680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9357.290412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9357.300536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.306234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.312227] Lustre: Skipped 1 previous similar message [ 9359.172089] Lustre: server umount lustre-MDT0000 complete [ 9361.120934] LustreError: 329010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307058 with bad export cookie 14495092154900972487 [ 9361.122979] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.129718] LustreError: 329010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9361.312841] Lustre: server umount lustre-MDT0001 complete [ 9368.900278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.068842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9369.096425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9369.206268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.594754] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9376.078875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9376.152433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9376.170492] Lustre: 331615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9376.310414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9377.979345] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9381.122401] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9383.743536] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9389.348023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9389.448629] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9389.453033] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9389.456918] LustreError: 333498:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9389.460382] LustreError: 333498:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9389.463539] LustreError: 333498:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9389.491750] Lustre: server umount lustre-OST0000 complete [ 9389.493137] LustreError: 333498:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9395.246391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9395.358978] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9395.485929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.264251] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9401.914794] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9405.535865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9406.122767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9406.175523] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9406.226635] Lustre: lustre-OST0001: new disk, initializing [ 9406.233556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9406.237454] Lustre: 335036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9406.285360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9409.864673] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9411.109946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9411.117047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9411.187635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9414.354692] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9416.167884] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9416.168425] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9416.175622] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9416.177025] LustreError: Skipped 1 previous similar message [ 9416.181225] Lustre: Skipped 1 previous similar message [ 9418.073096] Lustre: server umount lustre-OST0001 complete [ 9420.770176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9420.773931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.787794] Lustre: Skipped 1 previous similar message [ 9420.789794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9421.284772] LustreError: 334954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9421.294301] LustreError: 334954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9426.289278] Lustre: server umount lustre-OST0000 complete [ 9431.526602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9431.533703] Lustre: Skipped 1 previous similar message [ 9431.535848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9431.540907] Lustre: Skipped 3 previous similar messages [ 9434.451717] Lustre: server umount lustre-MDT0000 complete [ 9436.495490] LustreError: 331606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307133 with bad export cookie 14495092154900973040 [ 9436.498941] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.499427] LustreError: 331606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9436.658724] Lustre: server umount lustre-MDT0001 complete [ 9442.048399] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9443.600411] Key type lgssc unregistered [ 9443.767190] LNet: 336989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9443.774139] LNetError: 336989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9443.786934] LNet: Removed LNI 192.168.202.155@tcp [ 9444.343072] Key type .llcrypt unregistered [ 9444.345782] Key type ._llcrypt unregistered [ 9484.296399] Key type ._llcrypt registered [ 9484.297706] Key type .llcrypt registered [ 9484.346556] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 9491.842634] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9492.428995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9492.440352] alg: No test for adler32 (adler32-zlib) [ 9493.342829] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9493.458902] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9495.064234] Key type lgssc registered [ 9495.594975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9513.296501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9513.318067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.454554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9514.471130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9514.512412] Lustre: lustre-MDT0000: new disk, initializing [ 9514.543173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9514.550808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9516.695124] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9522.802896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.865854] Lustre: 341893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9522.894894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9522.899240] Lustre: Skipped 1 previous similar message [ 9522.938095] Lustre: lustre-MDT0001: new disk, initializing [ 9522.982744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9522.999466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9523.004355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9525.408469] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9530.864331] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9533.334790] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9536.826464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9536.977467] Lustre: lustre-OST0000: new disk, initializing [ 9536.980778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9536.986561] Lustre: 343864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9537.039982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9537.746145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9537.754832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9537.818221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9539.871112] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9545.106331] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9547.437751] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9547.544777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9549.719111] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9549.840250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9553.378440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9553.379198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.390479] LustreError: Skipped 1 previous similar message [ 9553.406182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9553.410590] Lustre: Skipped 1 previous similar message [ 9557.123628] Lustre: server umount lustre-OST0000 complete [ 9563.619623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.625978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9563.629652] Lustre: Skipped 2 previous similar messages [ 9563.636901] Lustre: Skipped 1 previous similar message [ 9565.581316] Lustre: server umount lustre-MDT0000 complete [ 9567.634537] LustreError: 341886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307264 with bad export cookie 4026082740551216363 [ 9567.637103] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9567.640309] LustreError: 341886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9567.825637] Lustre: server umount lustre-MDT0001 complete [ 9571.531544] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:47:49 (1789307269) [ 9575.289363] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:47:52 (1789307272) [ 9579.468632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9579.794852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.985593] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9585.126318] LustreError: 346615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.138545] LustreError: 346615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9586.452455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.666599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9588.875467] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9592.843098] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9595.499939] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9598.953811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9599.115809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9601.940422] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9605.391491] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9615.066651] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9615.847042] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9616.482036] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9617.151296] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9617.926786] Lustre: Failing over lustre-MDT0000 [ 9618.031754] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping) [ 9618.102394] Lustre: server umount lustre-MDT0000 complete [ 9622.497175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9622.501233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.506662] LustreError: 347343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.130133] LustreError: 347343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.135673] LustreError: 347343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9631.973869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9632.033477] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.169381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9632.199896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9633.381930] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9634.414609] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9637.359164] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9637.380984] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9637.414782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9640.164321] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9640.882865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9643.016751] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9644.081160] Lustre: Failing over lustre-MDT0000 [ 9644.205452] Lustre: server umount lustre-MDT0000 complete [ 9647.590257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9647.594174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.598096] LustreError: 346611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.605970] Lustre: Skipped 4 previous similar messages [ 9647.625721] LustreError: 346611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9652.706236] LustreError: 346616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.716648] LustreError: 346616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9658.674096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.763295] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9658.930035] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9658.988953] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9661.083848] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9663.976684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9663.979854] Lustre: Skipped 2 previous similar messages [ 9663.997929] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9664.018588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9666.969733] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9667.702305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9674.210214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9674.215683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9674.217381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9674.221308] Lustre: Skipped 2 previous similar messages [ 9674.224730] Lustre: Skipped 1 previous similar message [ 9676.639219] Lustre: server umount lustre-MDT0000 complete [ 9678.505893] LustreError: 349386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307375 with bad export cookie 4026082740551219058 [ 9678.506767] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9678.510675] LustreError: 349386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9678.647983] Lustre: server umount lustre-MDT0001 complete [ 9690.925285] Lustre: server umount lustre-OST0000 complete [ 9698.072474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.300861] LustreError: 352945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.311094] LustreError: 352945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9698.338436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.343296] Lustre: Skipped 1 previous similar message [ 9700.076520] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9703.953023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.789472] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9709.003958] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9711.429458] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9714.274516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9716.800329] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9719.860176] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9723.881185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9728.995607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9728.995667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9728.998485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9729.005732] Lustre: Skipped 1 previous similar message [ 9729.009467] Lustre: Skipped 1 previous similar message [ 9732.901945] Lustre: server umount lustre-OST0000 complete [ 9739.233802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9739.236684] Lustre: Skipped 2 previous similar messages [ 9740.793887] Lustre: server umount lustre-MDT0000 complete [ 9742.805927] LustreError: 354784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307439 with bad export cookie 4026082740551219807 [ 9742.814776] LustreError: 354784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9742.816213] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9742.982690] Lustre: server umount lustre-MDT0001 complete [ 9748.007419] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9749.352991] Key type lgssc unregistered [ 9749.524563] LNet: 356474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9749.528264] LNetError: 356474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9749.545200] LNet: Removed LNI 192.168.202.155@tcp [ 9750.014443] Key type .llcrypt unregistered [ 9750.016453] Key type ._llcrypt unregistered [ 9761.711761] Key type ._llcrypt registered [ 9761.713231] Key type .llcrypt registered [ 9761.764305] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9762.097025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9762.110057] alg: No test for adler32 (adler32-zlib) [ 9762.993730] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9763.101849] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9764.696346] Key type lgssc registered [ 9765.237097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9768.690396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9768.697901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.943153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.847593] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9775.640581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.824507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9777.586987] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9781.397624] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9784.165145] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9787.026387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9787.216596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9789.325819] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9792.040484] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9795.436847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9806.817164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9806.819709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.825711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9810.916221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9810.917021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9810.926693] Lustre: Skipped 1 previous similar message [ 9810.940140] Lustre: Skipped 2 previous similar messages [ 9812.919785] Lustre: server umount lustre-MDT0000 complete [ 9814.677339] LustreError: 358699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307511 with bad export cookie 5127325823793103926 [ 9814.680893] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.681974] LustreError: 358699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9814.865474] Lustre: server umount lustre-MDT0001 complete [ 9826.867505] Lustre: server umount lustre-OST0000 complete [ 9831.574830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.752193] LustreError: 361694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.758251] LustreError: 361694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9831.775041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.330529] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9836.642824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.770109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9838.272410] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9840.937214] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9842.894594] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9845.085721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9845.198767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9847.058257] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9849.550293] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9856.487094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9867.744870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9867.746982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.750670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9871.841774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9871.842654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.844418] Lustre: Skipped 1 previous similar message [ 9871.848933] Lustre: Skipped 1 previous similar message [ 9873.111633] Lustre: server umount lustre-MDT0000 complete [ 9874.313150] LustreError: 364394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307571 with bad export cookie 5127325823793105025 [ 9874.316219] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9874.317709] LustreError: 364394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9874.426995] Lustre: server umount lustre-MDT0001 complete [ 9885.913681] Lustre: server umount lustre-OST0000 complete [ 9891.448576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.605234] LustreError: 365620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.647735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.084028] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9895.947378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.430320] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9899.889829] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9901.827341] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9904.165945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.567502] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9909.269431] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9916.584480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9922.018193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9922.021449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.024883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9922.026680] Lustre: Skipped 2 previous similar messages [ 9922.770218] Lustre: server umount lustre-OST0000 complete [ 9927.138104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9927.140588] LustreError: Skipped 1 previous similar message [ 9930.134173] Lustre: server umount lustre-MDT0000 complete [ 9931.355537] LustreError: 365602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307628 with bad export cookie 5127325823793106033 [ 9931.358271] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9931.359920] LustreError: 365602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9931.468850] Lustre: server umount lustre-MDT0001 complete [ 9935.120165] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9936.159341] Key type lgssc unregistered [ 9936.285341] LNet: 369169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9936.287746] LNetError: 369169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9936.295314] LNet: Removed LNI 192.168.202.155@tcp [ 9936.607112] Key type .llcrypt unregistered [ 9936.608290] Key type ._llcrypt unregistered [ 9945.810900] Key type ._llcrypt registered [ 9945.811861] Key type .llcrypt registered [ 9945.848932] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9946.229193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9946.241948] alg: No test for adler32 (adler32-zlib) [ 9947.102125] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9947.193624] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9948.776161] Key type lgssc registered [ 9949.158462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9951.831292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9951.838351] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9951.841694] LustreError: 370582:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9951.850457] LustreError: 370582:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9951.852996] LustreError: 370582:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9951.855399] LustreError: 370582:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9951.858097] LustreError: 370582:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9954.425019] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9954.427581] LustreError: 370979:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9954.436452] LustreError: 370979:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9954.439353] LustreError: 370979:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9954.442461] LustreError: 370979:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9954.446592] LustreError: 370979:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9961.068039] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9962.138303] Key type lgssc unregistered [ 9962.275404] LNet: 371979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9962.278921] LNetError: 371979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9962.289273] LNet: Removed LNI 192.168.202.155@tcp [ 9962.592394] Key type .llcrypt unregistered [ 9962.594088] Key type ._llcrypt unregistered [ 9968.632034] Key type ._llcrypt registered [ 9968.633297] Key type .llcrypt registered [ 9968.671534] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 9975.090238] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9975.466728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9975.510931] alg: No test for adler32 (adler32-zlib) [ 9976.367360] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [ 9976.461352] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9978.040698] Key type lgssc registered [ 9978.380618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9989.587641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9989.601378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.681141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9990.691892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9990.724941] Lustre: lustre-MDT0000: new disk, initializing [ 9990.745679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.750961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9992.085677] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9996.341937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9996.383575] Lustre: 375105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9996.406611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9996.408400] Lustre: Skipped 1 previous similar message [ 9996.441930] Lustre: lustre-MDT0001: new disk, initializing [ 9996.474935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9996.488044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9996.491535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9997.811168] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10001.593064] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10003.443903] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10005.745868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.860973] Lustre: lustre-OST0000: new disk, initializing [10005.862943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10005.866399] Lustre: 377071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10005.899640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10007.919958] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10008.565272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10008.568522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10008.598163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10011.583481] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10013.028360] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10013.101800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10014.418645] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10014.492440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10018.786376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10018.786563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.789136] LustreError: Skipped 1 previous similar message [10018.789849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10018.792655] Lustre: Skipped 1 previous similar message [10021.209120] Lustre: server umount lustre-OST0000 complete [10023.905938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.906694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.910308] Lustre: Skipped 1 previous similar message [10023.913943] Lustre: Skipped 2 previous similar messages [10028.765863] Lustre: server umount lustre-MDT0000 complete [10030.079579] LustreError: 375898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307727 with bad export cookie 13238152601819876870 [10030.083129] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10030.084389] LustreError: 375898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10030.221667] Lustre: server umount lustre-MDT0001 complete [10033.023793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10033.582661] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:55:31 (1789307731) [10035.848737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.987439] LustreError: 379528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.992583] LustreError: 379528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10036.009709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10037.409992] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10040.585148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.742282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10042.115691] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10044.654949] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10046.501605] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10048.808514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.936355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10051.150892] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10053.751401] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10056.252513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10056.319268] Lustre: lustre-OST0001: new disk, initializing [10056.321293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10056.324753] Lustre: 382363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10058.632823] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10059.017690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10059.021266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10059.047923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10062.614331] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10067.428569] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.433898] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10067.436547] Lustre: Skipped 1 previous similar message [10070.660939] Lustre: server umount lustre-OST0001 complete [10072.551429] LustreError: 381518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10072.556904] LustreError: 381518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10077.668783] LustreError: 382785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.674233] LustreError: 382785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10082.788408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.788411] LustreError: 382785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.789178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10082.789406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10082.789410] Lustre: Skipped 1 previous similar message [10082.792355] Lustre: Skipped 2 previous similar messages [10082.797766] LustreError: 382785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10087.905787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10087.907225] LustreError: 381351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.908216] Lustre: Skipped 2 previous similar messages [10087.914299] LustreError: 381351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10097.120103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10097.175374] Lustre: server umount lustre-OST0000 complete [10102.240838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10102.244423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.250637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.253159] Lustre: Skipped 2 previous similar messages [10104.661436] Lustre: server umount lustre-MDT0000 complete [10105.920503] LustreError: 379511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307803 with bad export cookie 13238152601819877752 [10105.923981] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10105.925700] LustreError: 379511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10112.114948] Lustre: server umount lustre-MDT0001 complete [10115.612842] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10116.675934] Key type lgssc unregistered [10116.809382] LNet: 384348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10116.812289] LNetError: 384348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10116.821344] LNet: Removed LNI 192.168.202.155@tcp [10117.154918] Key type .llcrypt unregistered [10117.156442] Key type ._llcrypt unregistered [10135.567302] Key type ._llcrypt registered [10135.568378] Key type .llcrypt registered [10135.605926] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10135.902777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10135.918839] alg: No test for adler32 (adler32-zlib) [10136.776734] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10136.874461] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10138.456192] Key type lgssc registered [10138.876356] Lustre: Echo OBD driver; http://www.lustre.org/ [10141.736319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10141.744898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10142.828788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10142.836798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10142.893971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10144.379484] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10147.633267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10147.685311] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10147.698723] Lustre: 387119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10147.836930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10149.374159] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10152.159715] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10154.256549] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10157.537727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.538504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.542968] Lustre: Skipped 1 previous similar message [10161.246149] Lustre: server umount lustre-MDT0000 complete [10162.587277] LustreError: 387111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307859 with bad export cookie 14526195690071482341 [10162.590400] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10162.592032] LustreError: 387111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10162.709775] Lustre: server umount lustre-MDT0001 complete [10166.145511] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:57:43 (1789307863) [10168.565379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.749299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10170.182402] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10173.302981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.458904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10175.060815] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10177.958562] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.135993] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10182.588364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10182.674608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10182.743950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10184.797439] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10187.443766] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10192.333754] Lustre: Permanently deactivating lustre-OST0000 [10192.340772] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10201.071428] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10202.409931] LustreError: 392757:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10202.413325] LustreError: 392757:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10202.415397] LustreError: 392757:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10203.072066] LustreError: 392805:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10203.075130] LustreError: 392805:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10209.884429] Lustre: server umount lustre-OST0000 complete [10210.887743] LustreError: 393054:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10210.890036] LustreError: 393054:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10213.857759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.858429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10213.862115] Lustre: Skipped 1 previous similar message [10213.866046] Lustre: Skipped 2 previous similar messages [10218.977056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10218.979829] Lustre: Skipped 1 previous similar message [10221.051685] Lustre: server umount lustre-MDT0000 complete [10222.439571] LustreError: 389609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307919 with bad export cookie 14526195690071482908 [10222.442779] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10222.444549] LustreError: 389609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10232.852848] Lustre: server umount lustre-MDT0001 complete [10235.957973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.429838] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10238.383700] LustreError: 394210:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10240.548705] Lustre: 394549:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.155@tcp [10241.240344] Lustre: server umount MGS complete [10245.273531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.454385] LustreError: 395271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.459808] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10245.460969] LustreError: 395271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10245.463464] Lustre: Skipped 1 previous similar message [10245.484558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10246.827887] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10249.741708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.844459] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10251.257687] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10253.892899] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10255.710498] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10256.348496] Lustre: Permanently reactivating lustre-OST0000 [10256.354221] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10256.356431] Lustre: Skipped 2 previous similar messages [10282.976301] LustreError: 397099:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10286.350890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.456147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10286.458605] Lustre: Skipped 1 previous similar message [10288.320435] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10290.752216] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10302.944771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10302.946813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.950394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10303.971449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10308.429534] Lustre: server umount lustre-OST0000 complete [10311.648908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10311.652435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10311.657735] Lustre: Skipped 1 previous similar message [10311.659451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.660880] Lustre: Skipped 2 previous similar messages [10315.864839] Lustre: server umount lustre-MDT0000 complete [10317.238137] LustreError: 395251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308014 with bad export cookie 14526195690071483944 [10317.241432] LustreError: 395251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10317.241685] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10317.353506] Lustre: server umount lustre-MDT0001 complete [10321.096869] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10322.169973] Key type lgssc unregistered [10322.304347] LNet: 399989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10322.307278] LNetError: 399989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10322.318339] LNet: Removed LNI 192.168.202.155@tcp [10322.659272] Key type .llcrypt unregistered [10322.660534] Key type ._llcrypt unregistered [10328.887992] Key type ._llcrypt registered [10328.888949] Key type .llcrypt registered [10328.925993] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [10335.301352] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10335.767340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10335.775425] alg: No test for adler32 (adler32-zlib) [10336.637625] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10336.733342] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10338.320112] Key type lgssc registered [10338.738203] Lustre: Echo OBD driver; http://www.lustre.org/ [10349.842164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.847510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.938929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10350.951275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10350.990444] Lustre: lustre-MDT0000: new disk, initializing [10351.012068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10351.019152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10352.481804] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10356.719541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.770963] Lustre: 403115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10356.799233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10356.800759] Lustre: Skipped 1 previous similar message [10356.838437] Lustre: lustre-MDT0001: new disk, initializing [10356.875722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10356.888662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10356.892790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10358.286019] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10362.186987] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10363.973754] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10366.184808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10366.274764] Lustre: lustre-OST0000: new disk, initializing [10366.277182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10366.280776] Lustre: 405093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10366.308386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10368.175427] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10371.831793] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10373.114846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10373.118727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10373.151954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10373.211942] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10373.285852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10374.550980] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10374.623593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10378.211492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.211967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10378.214588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10378.215877] Lustre: Skipped 1 previous similar message [10378.218876] LustreError: Skipped 1 previous similar message [10381.328852] Lustre: server umount lustre-OST0000 complete [10382.816829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10382.819484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.824146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10382.825968] Lustre: Skipped 1 previous similar message [10388.448983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10388.450738] Lustre: Skipped 3 previous similar messages [10388.691791] Lustre: server umount lustre-MDT0000 complete [10389.923100] LustreError: 403108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308087 with bad export cookie 12018781793827637209 [10389.926088] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.928852] LustreError: 403108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10390.034296] Lustre: server umount lustre-MDT0001 complete [10392.742858] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10393.347083] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:01:31 (1789308091) [10395.529374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10395.700816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10397.030265] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10399.716268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.831954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10401.045275] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10403.442833] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10405.224900] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10407.453899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10409.379291] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10411.666798] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10412.282556] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10422.958529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10425.313779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10425.315991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10425.319611] Lustre: Skipped 1 previous similar message [10425.321128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10430.416228] Lustre: server umount lustre-OST0000 complete [10435.552903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10435.555203] LustreError: Skipped 1 previous similar message [10435.556724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.560261] Lustre: Skipped 1 previous similar message [10435.562639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10435.565147] Lustre: Skipped 2 previous similar messages [10437.777511] Lustre: server umount lustre-MDT0000 complete [10439.010212] LustreError: 409391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308136 with bad export cookie 12018781793827638091 [10439.013612] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.014073] LustreError: 409391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10439.135537] Lustre: server umount lustre-MDT0001 complete [10442.566228] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10443.549505] Key type lgssc unregistered [10443.679305] LNet: 411293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10443.681348] LNetError: 411293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10443.689327] LNet: Removed LNI 192.168.202.155@tcp [10443.982163] Key type .llcrypt unregistered [10443.983140] Key type ._llcrypt unregistered [10451.471882] Key type ._llcrypt registered [10451.472934] Key type .llcrypt registered [10451.510617] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10451.615307] Key type .llcrypt unregistered [10451.617044] Key type ._llcrypt unregistered [10458.410967] Key type ._llcrypt registered [10458.412395] Key type .llcrypt registered [10458.450337] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10458.938445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10458.943580] alg: No test for adler32 (adler32-zlib) [10459.813901] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10459.904442] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10461.488179] Key type lgssc registered [10461.889664] Lustre: Echo OBD driver; http://www.lustre.org/ [10464.662679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10464.671584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.808846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10467.096149] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10469.714016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.822945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10471.659798] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10473.961265] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10474.979392] LustreError: 413471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10474.982585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10474.985196] LustreError: 413471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10476.042736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.175329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10477.581724] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10485.216926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10485.220728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.225646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10486.754211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10486.754634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10486.756145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.513838] Lustre: server umount lustre-OST0000 complete [10491.873334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10491.873905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.874530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.874533] Lustre: Skipped 1 previous similar message [10491.880893] Lustre: Skipped 1 previous similar message [10496.993182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.994906] Lustre: Skipped 2 previous similar messages [10497.880276] Lustre: server umount lustre-MDT0000 complete [10500.697219] LustreError: 413454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308197 with bad export cookie 17114664419406250313 [10500.700160] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10500.700975] LustreError: 413454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10500.820188] Lustre: server umount lustre-MDT0001 complete [10504.745100] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10505.844782] Key type lgssc unregistered [10505.988319] LNet: 416620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10505.990969] LNetError: 416620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10505.999282] LNet: Removed LNI 192.168.202.155@tcp [10506.322561] Key type .llcrypt unregistered [10506.324102] Key type ._llcrypt unregistered [10516.348602] Key type ._llcrypt registered [10516.349551] Key type .llcrypt registered [10516.386107] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10516.795678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10516.820321] alg: No test for adler32 (adler32-zlib) [10517.681878] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10517.772639] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10519.352119] Key type lgssc registered [10519.736146] Lustre: Echo OBD driver; http://www.lustre.org/ [10522.344142] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10522.350164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10537.760423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10542.880398] LustreError: 417982:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.155@tcp: failed processing log, type 4: rc = -110 [10569.504225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10571.410606] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10573.812524] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10576.097570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.248358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10576.271225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10577.493532] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10580.349404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.479234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10581.836805] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10584.310755] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10586.167636] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10589.666514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10589.667569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10589.670684] Lustre: Skipped 1 previous similar message [10594.570890] Lustre: server umount lustre-OST0000 complete [10598.882120] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10598.884302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.889066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10598.890699] Lustre: Skipped 1 previous similar message [10599.905114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10599.905547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10602.006983] Lustre: server umount lustre-MDT0000 complete [10603.229046] LustreError: 419049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308300 with bad export cookie 2449670535226865353 [10603.232251] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.232992] LustreError: 419049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10603.346189] Lustre: server umount lustre-MDT0001 complete [10606.742833] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10607.631370] Key type lgssc unregistered [10607.752362] LNet: 421550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10607.754334] LNetError: 421550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10607.762260] LNet: Removed LNI 192.168.202.155@tcp [10608.046216] Key type .llcrypt unregistered [10608.047147] Key type ._llcrypt unregistered [10616.815917] Key type ._llcrypt registered [10616.816905] Key type .llcrypt registered [10616.853701] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10617.254790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10617.268160] alg: No test for adler32 (adler32-zlib) [10618.126719] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10618.215898] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10619.792159] Key type lgssc registered [10620.172810] Lustre: Echo OBD driver; http://www.lustre.org/ [10622.802799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10622.807919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.943743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10625.242018] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10627.977309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.120241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10629.488419] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10631.815517] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10633.589140] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10635.880343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.997878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10637.871137] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10640.210471] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10649.318667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10653.664893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10653.667285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.671173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.689403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.690196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.695289] Lustre: Skipped 1 previous similar message [10655.851420] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping) [10655.853964] Lustre: Skipped 2 previous similar messages [10657.683816] Lustre: server umount lustre-MDT0000 complete [10658.415192] LustreError: 422919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.55@tcp arrived at 1789308355 with bad export cookie 7774033382849174973 [10658.418984] LustreError: 422919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10659.025078] LustreError: 422921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308356 with bad export cookie 7774033382849174322 [10659.028633] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10659.028941] LustreError: 422921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10659.808940] LustreError: 422938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.809490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.813775] LustreError: 422938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10659.818604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10664.929038] LustreError: 422939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.929332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10664.934136] LustreError: 422939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10665.173143] Lustre: server umount lustre-OST0000 complete [10667.623712] Lustre: server umount lustre-MDT0001 complete [10671.259212] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10672.301700] Key type lgssc unregistered [10672.428447] LNet: 426638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10672.430983] LNetError: 426638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10672.441337] LNet: Removed LNI 192.168.202.155@tcp [10672.757293] Key type .llcrypt unregistered [10672.758339] Key type ._llcrypt unregistered [10681.493020] Key type ._llcrypt registered [10681.493916] Key type .llcrypt registered [10681.530292] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10681.941455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10681.948187] alg: No test for adler32 (adler32-zlib) [10682.793393] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10682.874587] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10684.448148] Key type lgssc registered [10684.771862] Lustre: Echo OBD driver; http://www.lustre.org/ [10687.150278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10687.156441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.284210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.449204] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10692.090341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.204108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10693.443944] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10695.646973] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10697.322963] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10699.525289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10699.647325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10701.529709] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10703.779415] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10713.004161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10718.179539] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10718.179905] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.187682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10718.189492] Lustre: Skipped 1 previous similar message [10719.341479] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [10721.297558] Lustre: server umount lustre-MDT0001 complete [10723.297965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10723.297964] LustreError: 428026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10723.300491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.305138] LustreError: 428026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10723.308588] Lustre: Skipped 1 previous similar message [10723.309102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10728.417205] LustreError: 428025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.417761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10729.038792] Lustre: server umount lustre-OST0000 complete [10730.413100] Lustre: server umount lustre-MDT0000 complete [10735.219539] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10736.259747] Key type lgssc unregistered [10736.398385] LNet: 431725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10736.400858] LNetError: 431725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10736.413309] LNet: Removed LNI 192.168.202.155@tcp [10736.730535] Key type .llcrypt unregistered [10736.731702] Key type ._llcrypt unregistered [10744.797306] Key type ._llcrypt registered [10744.798548] Key type .llcrypt registered [10744.832865] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [10750.623962] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10750.915643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10750.953457] alg: No test for adler32 (adler32-zlib) [10751.794213] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10751.873820] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10753.448167] Key type lgssc registered [10753.771036] Lustre: Echo OBD driver; http://www.lustre.org/ [10765.211330] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10770.255353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10770.260207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.334208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10771.343860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10771.379953] Lustre: lustre-MDT0000: new disk, initializing [10771.401652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.407901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10772.646422] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10778.112783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.202294] Lustre: lustre-OST0000: new disk, initializing [10778.204466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10778.206501] Lustre: Skipped 1 previous similar message [10778.208451] Lustre: 436729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10778.228686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10778.823097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10778.827390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10778.838090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10780.247473] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10783.806670] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10789.252924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.291430] Lustre: 435708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10789.314786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10789.354901] Lustre: lustre-MDT0001: new disk, initializing [10789.396533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.408067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10789.410468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10790.723429] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10794.224937] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10794.295248] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10795.642409] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10796.739289] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [10800.745585] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [10801.634496] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10801.634762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.638654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10801.641633] Lustre: Skipped 1 previous similar message [10804.434054] Lustre: server umount lustre-MDT0001 complete [10811.874025] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.878047] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10816.993190] LustreError: 435719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.993615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.004072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10817.005835] Lustre: Skipped 1 previous similar message [10822.113204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10822.113649] LustreError: 435718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10822.794260] Lustre: server umount lustre-OST0000 complete [10830.262062] Lustre: server umount lustre-MDT0000 complete [10834.755415] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10835.842430] Key type lgssc unregistered [10835.967382] LNet: 440479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10835.970156] LNetError: 440479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10835.976285] LNet: Removed LNI 192.168.202.155@tcp [10836.319606] Key type .llcrypt unregistered [10836.320799] Key type ._llcrypt unregistered [10849.115863] Key type ._llcrypt registered [10849.116877] Key type .llcrypt registered [10849.160188] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10849.270219] Key type .llcrypt unregistered [10849.271513] Key type ._llcrypt unregistered [10855.726489] Key type ._llcrypt registered [10855.727464] Key type .llcrypt registered [10855.766974] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10856.089875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10856.112089] alg: No test for adler32 (adler32-zlib) [10856.989369] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [10857.087648] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10858.672129] Key type lgssc registered [10859.046330] Lustre: Echo OBD driver; http://www.lustre.org/ [10868.000120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10868.005533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.088951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10869.099892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10869.144841] Lustre: lustre-MDT0000: new disk, initializing [10869.166260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10869.173463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10870.602577] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10874.865258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.910241] Lustre: 443816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10874.934281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10874.936156] Lustre: Skipped 1 previous similar message [10874.973101] Lustre: lustre-MDT0001: new disk, initializing [10875.010405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10875.024858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10875.028571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10876.475096] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10880.263689] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10882.305733] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10884.673832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.771842] Lustre: lustre-OST0000: new disk, initializing [10884.773487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10884.775885] Lustre: 445784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.803170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10886.628175] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10890.254346] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10891.257107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10891.260671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10891.286630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10900.960843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10900.962928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10900.966312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10901.473562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10901.474904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.475822] Lustre: Skipped 1 previous similar message [10901.479271] Lustre: Skipped 1 previous similar message [10903.637844] Lustre: server umount lustre-MDT0000 complete [10904.768167] LustreError: 443811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308601 with bad export cookie 12181932107890336504 [10904.771114] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.772649] LustreError: 443811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10904.882342] Lustre: server umount lustre-MDT0001 complete [10916.312272] Lustre: server umount lustre-OST0000 complete [10919.885196] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:10:17 (1789308617) [10922.593917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.799250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10924.179181] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10926.962904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.078828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10928.457802] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10931.038489] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10932.994045] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10948.323196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.383183] Lustre: Found index 0 for lustre-OST0000, updating log [10948.435171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.119136] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [10952.230487] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10968.934325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10974.179630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10974.180094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10974.183826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10974.188233] Lustre: Skipped 1 previous similar message [10974.191883] Lustre: Skipped 2 previous similar messages [10976.015251] Lustre: server umount lustre-OST0000 complete [10988.597707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.604915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.606802] Lustre: Skipped 2 previous similar messages [10993.633698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10993.633776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10993.633882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10993.635700] Lustre: Skipped 1 previous similar message [10994.779515] Lustre: server umount lustre-MDT0000 complete [10995.998249] LustreError: 449655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308693 with bad export cookie 12181932107890337897 [10996.001116] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10996.002169] LustreError: 449655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10996.122638] Lustre: server umount lustre-MDT0001 complete [10998.808243] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:11:36 (1789308696) [11018.912359] LNet: 1 local NIs in recovery (showing 1): 192.168.202.155@tcp [11592.628262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.785545] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.826939] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11594.021467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11599.337631] LustreError: 454241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.567764] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [11603.436902] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.584073] LustreError: 454241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.623445] LustreError: 454241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11614.725578] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.897311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11615.269957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11622.565463] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [11635.384880] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11646.881907] LNet: 1 local NIs in recovery (showing 1): 192.168.202.155@tcp [11647.178134] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11669.678183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11669.988768] Lustre: Found index 0 for lustre-OST0000, updating log [11670.350820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11677.086308] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [11684.862237] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11687.048135] Lustre: server umount lustre-OST0000 complete [11691.620391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11691.630463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11691.634678] Lustre: Skipped 1 previous similar message [11692.641302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.658937] Lustre: Skipped 1 previous similar message [11694.690303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11694.700182] Lustre: Skipped 3 previous similar messages [11696.976835] Lustre: server umount lustre-MDT0000 complete [11697.764100] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.783908] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11700.756221] LustreError: 454221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309397 with bad export cookie 12181932107890338968 [11700.757952] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11700.768733] LustreError: 454221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11701.245922] Lustre: server umount lustre-MDT0001 complete [11720.609951] LNet: 1 local NIs in recovery (showing 1): 192.168.202.155@tcp [11732.482036] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [11744.702289] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [11784.379863] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:24:40 (1789309480) [11806.134324] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [11809.423119] Key type lgssc unregistered [11809.853266] LNet: 460855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11809.863643] LNetError: 460855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11932.778322] LNet: Removed LNI 192.168.202.155@tcp [11934.515272] Key type .llcrypt unregistered [11934.518799] Key type ._llcrypt unregistered [11955.949073] Key type ._llcrypt registered [11955.952352] Key type .llcrypt registered [11956.064565] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [11957.913933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11958.026998] alg: No test for adler32 (adler32-zlib) [11959.123274] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [11959.522857] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [11961.258464] Key type lgssc registered [11962.471172] Lustre: Echo OBD driver; http://www.lustre.org/ [11967.435591] LNet: Added LNI 192.168.202.155@tcp100 [8/256/0/180] [11968.802597] LNet: Added LNI 192.168.202.155@tcp101 [8/256/0/180] [11969.903600] LNet: Added LNI 192.168.202.155@tcp102 [8/256/0/180] [11970.902940] LNet: Added LNI 192.168.202.155@tcp103 [8/256/0/180] [11971.948785] LNet: Added LNI 192.168.202.155@tcp104 [8/256/0/180] [11973.105966] LNet: Added LNI 192.168.202.155@tcp105 [8/256/0/180] [11974.405149] LNet: Added LNI 192.168.202.155@tcp106 [8/256/0/180] [11975.684279] LNet: Added LNI 192.168.202.155@tcp107 [8/256/0/180] [11976.644208] LNet: Added LNI 192.168.202.155@tcp108 [8/256/0/180] [11977.352623] LNet: Added LNI 192.168.202.155@tcp109 [8/256/0/180] [11978.148314] LNet: Added LNI 192.168.202.155@tcp110 [8/256/0/180] [11979.435038] LNet: Added LNI 192.168.202.155@tcp111 [8/256/0/180] [11980.781327] LNet: Added LNI 192.168.202.155@tcp112 [8/256/0/180] [11982.651825] LNet: Added LNI 192.168.202.155@tcp113 [8/256/0/180] [11984.470945] LNet: Added LNI 192.168.202.155@tcp114 [8/256/0/180] [11986.130419] LNet: Added LNI 192.168.202.155@tcp115 [8/256/0/180] [11987.141168] LNet: Added LNI 192.168.202.155@tcp116 [8/256/0/180] [11988.089359] LNet: Added LNI 192.168.202.155@tcp117 [8/256/0/180] [11989.144403] LNet: Added LNI 192.168.202.155@tcp118 [8/256/0/180] [11990.346360] LNet: Added LNI 192.168.202.155@tcp119 [8/256/0/180] [11991.460489] LNet: Added LNI 192.168.202.155@tcp120 [8/256/0/180] [11992.477056] LNet: Added LNI 192.168.202.155@tcp121 [8/256/0/180] [11993.496679] LNet: Added LNI 192.168.202.155@tcp122 [8/256/0/180] [11994.529463] LNet: Added LNI 192.168.202.155@tcp123 [8/256/0/180] [11995.908673] LNet: Added LNI 192.168.202.155@tcp124 [8/256/0/180] [11997.081036] LNet: Added LNI 192.168.202.155@tcp125 [8/256/0/180] [11998.188248] LNet: Added LNI 192.168.202.155@tcp126 [8/256/0/180] [11999.203779] LNet: Added LNI 192.168.202.155@tcp127 [8/256/0/180] [12000.116478] LNet: Added LNI 192.168.202.155@tcp128 [8/256/0/180] [12001.110435] LNet: Added LNI 192.168.202.155@tcp129 [8/256/0/180] [12002.282232] LNet: Added LNI 192.168.202.155@tcp130 [8/256/0/180] [12003.370493] LNet: Added LNI 192.168.202.155@tcp131 [8/256/0/180] [12004.380659] LNet: Added LNI 192.168.202.155@tcp132 [8/256/0/180] [12005.281589] LNet: Added LNI 192.168.202.155@tcp133 [8/256/0/180] [12006.307768] LNet: Added LNI 192.168.202.155@tcp134 [8/256/0/180] [12007.226296] LNet: Added LNI 192.168.202.155@tcp135 [8/256/0/180] [12016.066273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12016.151975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.349314] Lustre: 463887:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.155@tcp: can't add NID 192.168.202.155@tcp130: rc = -75 [12017.506261] LNetError: 463913:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12017.537042] LNetError: Protocol error connecting to 192.168.202.155@tcp121 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12017.567062] LNetError: Skipped 5 previous similar messages [12017.696929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12017.775228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12017.888118] LNet: 4 local NIs in recovery (showing 4): 192.168.202.155@tcp127, 192.168.202.155@tcp129, 192.168.202.155@tcp121, 192.168.202.155@tcp128 [12017.912138] Lustre: lustre-MDT0000: new disk, initializing [12017.988737] Lustre: 463945:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.155@tcp: can't add NID 192.168.202.155@tcp130: rc = -75 [12018.000492] Lustre: 463945:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12018.024658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12018.059446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12018.595679] LNetError: 463920:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12018.609876] LNetError: 463920:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [12018.619472] LNetError: Protocol error connecting to 192.168.202.155@tcp at host 192.168.202.155:988: is it running a compatible version of Lustre? [12018.625149] LNetError: Skipped 2 previous similar messages [12020.993050] LNetError: 463911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12021.012721] LNetError: 463911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [12021.030325] LNetError: Protocol error connecting to 192.168.202.155@tcp127 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12021.038897] LNetError: Skipped 4 previous similar messages [12023.707913] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12032.302301] LNetError: 463908:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12032.318266] LNetError: Protocol error connecting to 192.168.202.155@tcp127 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12036.223319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.386869] Lustre: 463902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12036.453514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12036.456670] Lustre: Skipped 1 previous similar message [12036.539987] Lustre: lustre-MDT0001: new disk, initializing [12036.582476] Lustre: 464708:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.155@tcp: can't add NID 192.168.202.155@tcp130: rc = -75 [12036.591168] Lustre: 464708:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12036.704200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12036.752670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12036.769043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12042.239348] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12042.333198] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.155@tcp: can't add NID 192.168.202.155@tcp130: rc = -75 [12042.342253] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12048.674686] LNetError: 463921:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12048.700404] LNetError: Protocol error connecting to 192.168.202.155@tcp124 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12054.663293] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12056.875230] LNetError: 463907:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12056.887157] LNetError: 463907:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [12056.913547] LNetError: Protocol error connecting to 192.168.202.155@tcp123 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12056.930365] LNetError: Skipped 4 previous similar messages [12062.879057] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12072.975296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12073.485472] Lustre: lustre-OST0000: new disk, initializing [12073.487476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12073.493683] Lustre: 465881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12073.591789] Lustre: 465886:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.155@tcp: can't add NID 192.168.202.155@tcp130: rc = -75 [12073.603422] Lustre: 465886:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12073.691426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12074.292834] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12074.307697] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [12074.313849] LNetError: Protocol error connecting to 192.168.202.155@tcp124 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12074.322686] LNetError: Skipped 8 previous similar messages [12081.258418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12081.270728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12081.504439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12082.347913] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12094.318172] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12107.250050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12107.254043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.278607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.291871] Lustre: Skipped 1 previous similar message [12109.986480] Lustre: server umount lustre-OST0000 complete [12112.162284] LNetError: 463908:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.155 [12112.172028] LNetError: 463908:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [12112.195698] LNetError: Protocol error connecting to 192.168.202.155@tcp124 at host 192.168.202.155:988: is it running a compatible version of Lustre? [12112.210475] LNetError: Skipped 13 previous similar messages [12117.485836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.524369] Lustre: Skipped 1 previous similar message [12117.536131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.538674] Lustre: Skipped 1 previous similar message [12121.441606] Lustre: server umount lustre-MDT0000 complete [12125.611612] LustreError: 463895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789309822 with bad export cookie 3125506817018027685 [12125.623856] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12125.633564] LustreError: 463895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12126.382512] Lustre: server umount lustre-MDT0001 complete [12140.775934] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [12144.099827] Key type lgssc unregistered [12144.554214] LNet: 467796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12144.564887] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12144.573726] LNet: Removed LNI 192.168.202.155@tcp [12145.634038] LNet: Removed LNI 192.168.202.155@tcp100 [12145.636768] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12145.657081] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12145.672171] LNet: Removed LNI 192.168.202.155@tcp101 [12146.720363] LNet: Removed LNI 192.168.202.155@tcp102 [12146.722382] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12146.742360] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12146.756267] LNet: Removed LNI 192.168.202.155@tcp103 [12147.808195] LNet: Removed LNI 192.168.202.155@tcp104 [12148.832822] LNet: Removed LNI 192.168.202.155@tcp105 [12148.839729] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12148.857535] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12148.873823] LNet: Removed LNI 192.168.202.155@tcp106 [12149.920670] LNet: Removed LNI 192.168.202.155@tcp107 [12150.944339] LNet: Removed LNI 192.168.202.155@tcp108 [12152.032332] LNet: Removed LNI 192.168.202.155@tcp109 [12153.056195] LNet: Removed LNI 192.168.202.155@tcp110 [12153.063090] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12153.072942] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12153.088656] LNet: Removed LNI 192.168.202.155@tcp111 [12154.144645] LNet: Removed LNI 192.168.202.155@tcp112 [12155.232522] LNet: Removed LNI 192.168.202.155@tcp113 [12156.256363] LNet: Removed LNI 192.168.202.155@tcp114 [12157.280271] LNet: Removed LNI 192.168.202.155@tcp115 [12158.304793] LNet: Removed LNI 192.168.202.155@tcp116 [12159.328241] LNet: Removed LNI 192.168.202.155@tcp117 [12160.352596] LNet: Removed LNI 192.168.202.155@tcp118 [12161.380200] LNet: Removed LNI 192.168.202.155@tcp119 [12161.392986] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12161.401163] LNetError: 467796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12161.412353] LNet: Removed LNI 192.168.202.155@tcp120 [12162.464235] LNet: Removed LNI 192.168.202.155@tcp121 [12163.488201] LNet: Removed LNI 192.168.202.155@tcp122 [12164.513342] LNet: Removed LNI 192.168.202.155@tcp123 [12165.555549] LNet: Removed LNI 192.168.202.155@tcp124 [12166.624252] LNet: Removed LNI 192.168.202.155@tcp125 [12167.648285] LNet: Removed LNI 192.168.202.155@tcp126 [12168.672719] LNet: Removed LNI 192.168.202.155@tcp127 [12169.696239] LNet: Removed LNI 192.168.202.155@tcp128 [12170.720510] LNet: Removed LNI 192.168.202.155@tcp129 [12170.726172] LNet: Removed LNI 192.168.202.155@tcp130 [12170.736217] LNet: Removed LNI 192.168.202.155@tcp131 [12170.753420] LNet: Removed LNI 192.168.202.155@tcp132 [12170.761745] LNet: Removed LNI 192.168.202.155@tcp133 [12170.770334] LNet: Removed LNI 192.168.202.155@tcp134 [12170.793607] LNet: Removed LNI 192.168.202.155@tcp135 [12171.893269] Key type .llcrypt unregistered [12171.898653] Key type ._llcrypt unregistered [12200.385879] Key type ._llcrypt registered [12200.389739] Key type .llcrypt registered [12200.517981] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [12219.613133] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [12221.434053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12221.494786] alg: No test for adler32 (adler32-zlib) [12222.512151] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [12222.706273] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [12224.352501] Key type lgssc registered [12225.916529] Lustre: Echo OBD driver; http://www.lustre.org/ [12268.354304] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:32:44 (1789309964) [12291.282510] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [12294.936259] Key type lgssc unregistered [12295.490382] LNet: 471887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12295.513993] LNetError: 471887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12295.583473] LNet: Removed LNI 192.168.202.155@tcp [12296.903703] Key type .llcrypt unregistered [12296.905772] Key type ._llcrypt unregistered [12314.637986] Key type ._llcrypt registered [12314.649820] Key type .llcrypt registered [12314.811177] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [12315.645871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12315.872153] alg: No test for adler32 (adler32-zlib) [12317.154685] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [12317.511576] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [12319.328242] Key type lgssc registered [12320.605693] Lustre: Echo OBD driver; http://www.lustre.org/ [12331.528643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12331.560617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12332.950039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12332.986554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12333.094589] Lustre: lustre-MDT0000: new disk, initializing [12333.200110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12333.239454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12339.092631] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12351.583123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12351.762400] Lustre: 473095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12351.857391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12351.864170] Lustre: Skipped 1 previous similar message [12352.005682] Lustre: lustre-MDT0001: new disk, initializing [12352.210039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12352.292360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12352.321662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12358.702773] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12370.466311] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12377.698764] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12385.625761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12385.999550] Lustre: lustre-OST0000: new disk, initializing [12386.012278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12386.029927] Lustre: 475066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12386.187666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12393.199153] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12395.200778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12395.221557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12395.328751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12403.564629] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12408.491559] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing lnet_if_list [12416.388816] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12416.454593] LNet: Added LNI 192.168.202.155@tcp10 [8/256/0/180] [12426.101705] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12426.155799] LNet: Added LNI 192.168.202.155@tcp11 [8/256/0/180] [12431.025591] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12431.070738] LNet: Added LNI 192.168.202.155@tcp12 [8/256/0/180] [12436.914846] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12436.948292] LNet: Added LNI 192.168.202.155@tcp13 [8/256/0/180] [12442.922232] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12442.949548] LNet: Added LNI 192.168.202.155@tcp14 [8/256/0/180] [12447.703228] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12447.740726] LNet: Added LNI 192.168.202.155@tcp15 [8/256/0/180] [12453.268608] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12453.291945] LNet: Added LNI 192.168.202.155@tcp16 [8/256/0/180] [12458.693887] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12458.730621] LNet: Added LNI 192.168.202.155@tcp17 [8/256/0/180] [12465.041549] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12465.069671] LNet: Added LNI 192.168.202.155@tcp18 [8/256/0/180] [12471.158819] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12471.196461] LNet: Added LNI 192.168.202.155@tcp19 [8/256/0/180] [12477.682228] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12477.737968] LNet: Added LNI 192.168.202.155@tcp20 [8/256/0/180] [12483.694282] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12483.727238] LNet: Added LNI 192.168.202.155@tcp21 [8/256/0/180] [12491.054410] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12491.088571] LNet: Added LNI 192.168.202.155@tcp22 [8/256/0/180] [12497.908231] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12497.959081] LNet: Added LNI 192.168.202.155@tcp23 [8/256/0/180] [12504.749896] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12504.810322] LNet: Added LNI 192.168.202.155@tcp24 [8/256/0/180] [12510.962106] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12510.983082] LNet: Added LNI 192.168.202.155@tcp25 [8/256/0/180] [12517.407654] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12517.443831] LNet: Added LNI 192.168.202.155@tcp26 [8/256/0/180] [12524.697587] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12524.728789] LNet: Added LNI 192.168.202.155@tcp27 [8/256/0/180] [12531.641622] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12531.663896] LNet: Added LNI 192.168.202.155@tcp28 [8/256/0/180] [12538.353527] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12538.413773] LNet: Added LNI 192.168.202.155@tcp29 [8/256/0/180] [12544.686613] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12544.726326] LNet: Added LNI 192.168.202.155@tcp30 [8/256/0/180] [12551.052540] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12551.087458] LNet: Added LNI 192.168.202.155@tcp31 [8/256/0/180] [12558.354589] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12558.388773] LNet: Added LNI 192.168.202.155@tcp32 [8/256/0/180] [12566.313475] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12566.339753] LNet: Added LNI 192.168.202.155@tcp33 [8/256/0/180] [12573.510864] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12573.528897] LNet: Added LNI 192.168.202.155@tcp34 [8/256/0/180] [12580.364215] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12580.384952] LNet: Added LNI 192.168.202.155@tcp35 [8/256/0/180] [12587.963635] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12587.992533] LNet: Added LNI 192.168.202.155@tcp36 [8/256/0/180] [12595.338989] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12595.367106] LNet: Added LNI 192.168.202.155@tcp37 [8/256/0/180] [12603.141516] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12603.169041] LNet: Added LNI 192.168.202.155@tcp38 [8/256/0/180] [12610.701420] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12610.736530] LNet: Added LNI 192.168.202.155@tcp39 [8/256/0/180] [12618.919831] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12618.944769] LNet: Added LNI 192.168.202.155@tcp40 [8/256/0/180] [12627.149422] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12627.179829] LNet: Added LNI 192.168.202.155@tcp41 [8/256/0/180] [12635.319619] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12635.345525] LNet: Added LNI 192.168.202.155@tcp42 [8/256/0/180] [12643.519784] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12643.577812] LNet: Added LNI 192.168.202.155@tcp43 [8/256/0/180] [12651.616483] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12651.652054] LNet: Added LNI 192.168.202.155@tcp44 [8/256/0/180] [12661.069936] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12661.101564] LNet: Added LNI 192.168.202.155@tcp45 [8/256/0/180] [12669.234860] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12669.254235] LNet: Added LNI 192.168.202.155@tcp46 [8/256/0/180] [12677.072851] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12677.099945] LNet: Added LNI 192.168.202.155@tcp47 [8/256/0/180] [12685.190879] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12685.220117] LNet: Added LNI 192.168.202.155@tcp48 [8/256/0/180] [12693.463391] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12693.487395] LNet: Added LNI 192.168.202.155@tcp49 [8/256/0/180] [12701.714710] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12701.749705] LNet: Added LNI 192.168.202.155@tcp50 [8/256/0/180] [12731.367845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12731.368095] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.390486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12731.417143] Lustre: Skipped 1 previous similar message [12731.440757] Lustre: Skipped 2 previous similar messages [12734.699066] Lustre: server umount lustre-MDT0000 complete [12739.504950] LustreError: 473899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310436 with bad export cookie 15874715870747477760 [12739.519244] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.524544] LustreError: 473899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12740.153383] Lustre: server umount lustre-MDT0001 complete [12756.960165] Lustre: 472368:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310438/real 1789310438] req@00000000faa26e4b x1876227537423232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789310454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12757.008434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.034748] Lustre: Skipped 1 previous similar message [12760.163120] Lustre: 472368:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310441/real 1789310441] req@000000006cbf3228 x1876227537423360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789310457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12762.080425] Lustre: 472366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310443/real 1789310443] req@00000000c9fe09a7 x1876227537423488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789310459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12762.130898] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [12765.153718] Lustre: 472367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310446/real 1789310446] req@0000000002974171 x1876227537423744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789310462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12771.816969] Lustre: 472368:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789310452/real 1789310452] req@000000008e8a4259 x1876227537423872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789310468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12773.108975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12784.811110] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdc4e51fa0228d14f [12784.842301] Lustre: MGC192.168.202.155@tcp: Connection restored to 0@lo (at 0@lo) [12785.204363] LustreError: 475059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.235516] LustreError: 475059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12785.279935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12786.279952] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.306366] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12787.297314] LustreError: 475059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.039876] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12792.802618] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.934792] LustreError: 475059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.550820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12800.019703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12802.103565] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12804.732257] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12812.887896] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12820.882717] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12846.053973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12846.063181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.081460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.108374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.125091] Lustre: Skipped 2 previous similar messages [12849.841660] Lustre: server umount lustre-OST0000 complete [12856.290807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.309949] LustreError: Skipped 1 previous similar message [12856.322827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.354642] Lustre: Skipped 1 previous similar message [12856.377407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12860.222782] Lustre: server umount lustre-MDT0000 complete [12863.458495] LustreError: 499336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12863.487300] LustreError: 499336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12864.425758] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310561 with bad export cookie 15874715870747480399 [12864.434587] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12864.435732] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12864.777850] Lustre: server umount lustre-MDT0001 complete [12878.065874] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [12881.815511] Key type lgssc unregistered [12882.172653] LNet: 502133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12882.188164] LNetError: 502133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12882.194488] LNet: Removed LNI 192.168.202.155@tcp [12882.201633] LNet: Removed LNI 192.168.202.155@tcp10 [12882.205713] LNet: Removed LNI 192.168.202.155@tcp11 [12882.216270] LNet: Removed LNI 192.168.202.155@tcp12 [12882.218146] LNet: Removed LNI 192.168.202.155@tcp13 [12882.219943] LNet: Removed LNI 192.168.202.155@tcp14 [12882.243616] LNet: Removed LNI 192.168.202.155@tcp15 [12882.246644] LNet: Removed LNI 192.168.202.155@tcp16 [12882.251634] LNet: Removed LNI 192.168.202.155@tcp17 [12882.257294] LNet: Removed LNI 192.168.202.155@tcp18 [12882.260396] LNet: Removed LNI 192.168.202.155@tcp19 [12882.266941] LNet: Removed LNI 192.168.202.155@tcp20 [12882.270194] LNet: Removed LNI 192.168.202.155@tcp21 [12882.274820] LNet: Removed LNI 192.168.202.155@tcp22 [12882.276522] LNet: Removed LNI 192.168.202.155@tcp23 [12882.279451] LNet: Removed LNI 192.168.202.155@tcp24 [12882.282452] LNet: Removed LNI 192.168.202.155@tcp25 [12882.294036] LNet: Removed LNI 192.168.202.155@tcp26 [12882.299089] LNet: Removed LNI 192.168.202.155@tcp27 [12882.304841] LNet: Removed LNI 192.168.202.155@tcp28 [12882.317283] LNet: Removed LNI 192.168.202.155@tcp29 [12882.320469] LNet: Removed LNI 192.168.202.155@tcp30 [12882.323249] LNet: Removed LNI 192.168.202.155@tcp31 [12882.331324] LNet: Removed LNI 192.168.202.155@tcp32 [12882.339285] LNet: Removed LNI 192.168.202.155@tcp33 [12882.341126] LNet: Removed LNI 192.168.202.155@tcp34 [12882.346369] LNet: Removed LNI 192.168.202.155@tcp35 [12882.350402] LNet: Removed LNI 192.168.202.155@tcp36 [12882.355993] LNet: Removed LNI 192.168.202.155@tcp37 [12882.359087] LNet: Removed LNI 192.168.202.155@tcp38 [12882.376863] LNet: Removed LNI 192.168.202.155@tcp39 [12882.378521] LNet: Removed LNI 192.168.202.155@tcp40 [12882.393828] LNet: Removed LNI 192.168.202.155@tcp41 [12882.400545] LNet: Removed LNI 192.168.202.155@tcp42 [12882.404770] LNet: Removed LNI 192.168.202.155@tcp43 [12882.408374] LNet: Removed LNI 192.168.202.155@tcp44 [12882.412326] LNet: Removed LNI 192.168.202.155@tcp45 [12882.420362] LNet: Removed LNI 192.168.202.155@tcp46 [12882.426784] LNet: Removed LNI 192.168.202.155@tcp47 [12882.429201] LNet: Removed LNI 192.168.202.155@tcp48 [12882.432486] LNet: Removed LNI 192.168.202.155@tcp49 [12882.441888] LNet: Removed LNI 192.168.202.155@tcp50 [12883.796791] Key type .llcrypt unregistered [12883.798951] Key type ._llcrypt unregistered [12911.608559] Key type ._llcrypt registered [12911.622282] Key type .llcrypt registered [12911.809543] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [12928.656867] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [12930.380515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12930.405219] alg: No test for adler32 (adler32-zlib) [12931.477880] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [12931.708083] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [12933.458614] Key type lgssc registered [12934.763426] Lustre: Echo OBD driver; http://www.lustre.org/ [12970.169902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12970.193126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12971.427920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12971.458978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12971.562292] Lustre: lustre-MDT0000: new disk, initializing [12971.638879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12971.656938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12975.294646] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [12984.874992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12984.974110] Lustre: 505325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12985.021631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12985.024676] Lustre: Skipped 1 previous similar message [12985.080218] Lustre: lustre-MDT0001: new disk, initializing [12985.152048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12985.195455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12985.198832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12990.809622] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13000.114511] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13006.507226] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13016.121460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13016.308700] Lustre: *** cfs_fail_loc=90f, val=0*** [13016.383388] Lustre: lustre-OST0000: new disk, initializing [13016.387376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13016.398701] Lustre: 507345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13016.495243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13023.304183] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13023.796578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13023.813070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13023.979537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13034.260766] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13044.711963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.721816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13044.721964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13044.734886] Lustre: Skipped 1 previous similar message [13044.761879] Lustre: Skipped 1 previous similar message [13048.689416] Lustre: server umount lustre-OST0000 complete [13054.946613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.979290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13054.987173] Lustre: Skipped 1 previous similar message [13058.560136] Lustre: server umount lustre-MDT0000 complete [13062.247445] LustreError: 505318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789310759 with bad export cookie 11245233402741333214 [13062.259856] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13062.273184] LustreError: 505318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13062.520460] Lustre: server umount lustre-MDT0001 complete [13074.657599] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [13077.425331] Key type lgssc unregistered [13077.758746] LNet: 509253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13077.768540] LNetError: 509253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13078.827039] LNet: Removed LNI 192.168.202.155@tcp [13079.923106] Key type .llcrypt unregistered [13079.927298] Key type ._llcrypt unregistered [13101.256572] Key type ._llcrypt registered [13101.258216] Key type .llcrypt registered [13101.370506] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [13102.613634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13102.754043] alg: No test for adler32 (adler32-zlib) [13103.806234] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [13104.116968] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [13105.816204] Key type lgssc registered [13107.279814] Lustre: Echo OBD driver; http://www.lustre.org/ [13115.544891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13115.558209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13117.141834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13123.216169] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13127.662106] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.674759] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13132.773092] LustreError: 510520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13133.272299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13133.768253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13138.287252] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13146.798908] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13152.862279] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13160.176748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13160.578882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13166.632407] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13173.743123] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13189.056897] loop0: detected capacity change from 0 to 204800000 [13189.290553] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13189.305938] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13189.327373] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13197.275826] loop0: detected capacity change from 0 to 204800000 [13197.539849] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13197.555424] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13197.574734] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13204.290121] loop0: detected capacity change from 0 to 204800000 [13211.857202] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.121765] Lustre: fs15246-OST0000: new disk, initializing [13212.125734] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13212.132728] Lustre: 514113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13212.195353] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13218.747953] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13228.551450] loop1: detected capacity change from 0 to 204800000 [13234.488928] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.610332] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13234.723612] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13234.863507] Lustre: fs15246-MDT0000: new disk, initializing [13235.036316] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13235.066344] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13235.198452] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13236.258445] Lustre: 515235:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000538489e5 x1876228362520960/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789310949 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13240.546739] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13241.323461] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13241.336977] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13241.385386] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13265.890928] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.913567] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13271.015393] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13272.130974] Lustre: server umount fs15246-MDT0000 complete [13276.132669] LustreError: 510516:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13281.254468] LustreError: 510514:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13283.024877] Lustre: server umount fs15246-OST0000 complete [13301.217537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13301.232516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13301.234585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13301.249039] Lustre: Skipped 1 previous similar message [13302.671304] Lustre: server umount lustre-OST0000 complete [13307.875287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13307.888353] LustreError: Skipped 1 previous similar message [13307.897863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13307.926657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13307.932402] Lustre: Skipped 1 previous similar message [13311.972967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13311.972975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13313.738768] Lustre: server umount lustre-MDT0000 complete [13317.098499] LustreError: 511535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.137941] LustreError: 511535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13318.546039] LustreError: 510500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311015 with bad export cookie 1728107746490055272 [13318.550843] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13318.560279] LustreError: 510500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13318.918944] Lustre: server umount lustre-MDT0001 complete [13331.576480] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [13335.501544] Key type lgssc unregistered [13335.844770] LNet: 518335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13335.858166] LNetError: 518335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13335.879724] LNet: Removed LNI 192.168.202.155@tcp [13336.859199] Key type .llcrypt unregistered [13336.861254] Key type ._llcrypt unregistered [13358.164292] Key type ._llcrypt registered [13358.165819] Key type .llcrypt registered [13358.271666] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [13359.559429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13359.606424] alg: No test for adler32 (adler32-zlib) [13361.090496] Lustre: Lustre: Build Version: 2.17.57_106_gd81f50d [13361.435325] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [13363.176244] Key type lgssc registered [13364.918197] Lustre: Echo OBD driver; http://www.lustre.org/ [13411.382056] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [13423.900423] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [13466.650943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13466.700875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13468.085469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13468.120141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13468.187545] Lustre: lustre-MDT0000: new disk, initializing [13468.271218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13468.290882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13473.109466] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13485.493249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13485.620220] Lustre: 522801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13485.665404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13485.671410] Lustre: Skipped 1 previous similar message [13485.756275] Lustre: lustre-MDT0001: new disk, initializing [13485.880680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13485.934444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13485.946309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13490.771107] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13501.267290] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13507.663984] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13516.232792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13516.765977] Lustre: lustre-OST0000: new disk, initializing [13516.780854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13516.791193] Lustre: 524769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13516.937973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13523.060209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13523.066103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13523.274630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13523.967515] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [13535.299857] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13540.771481] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13540.969302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13546.580445] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13546.899977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13549.028474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13549.036448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.047854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13549.054756] Lustre: Skipped 1 previous similar message [13554.152127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.164449] Lustre: Skipped 1 previous similar message [13559.267320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13559.276233] Lustre: Skipped 1 previous similar message [13563.360149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13563.606820] Lustre: server umount lustre-OST0000 complete [13567.972537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13567.980737] LustreError: Skipped 1 previous similar message [13567.982856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13568.007272] Lustre: Skipped 1 previous similar message [13568.015840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13569.507339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13574.629324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13574.642059] Lustre: Skipped 3 previous similar messages [13582.305316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13582.494772] Lustre: server umount lustre-MDT0000 complete [13586.633824] LustreError: 523604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789311283 with bad export cookie 372976782593390990 [13586.642695] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13586.648525] LustreError: 523604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13587.057795] Lustre: server umount lustre-MDT0001 complete [13614.108436] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [13628.387609] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [13667.067434] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13418 sec ========= 10:56:03 (1789311363) [13669.106637] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:56:05 (1789311365) === [13672.887894] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:56:09 (1789311369) === [13718.219434] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [13721.135912] Key type lgssc unregistered [13721.568147] LNet: 531418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13721.578276] LNetError: 531418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13721.618344] LNet: Removed LNI 192.168.202.155@tcp [13722.935675] Key type .llcrypt unregistered [13722.937625] Key type ._llcrypt unregistered