[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 531660411 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006930] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009158] LSM: Security Framework initializing [ 0.010075] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012078] *** VALIDATE selinux *** [ 0.020832] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026573] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030144] *** VALIDATE tmpfs *** [ 0.031493] *** VALIDATE proc *** [ 0.032270] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035322] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037076] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043405] debug: unmapping init [mem 0xffffffffaa259000-0xffffffffaa260fff] [ 0.046106] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049019] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.050991] ... value mask: 0000ffffffffffff [ 0.051021] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053019] ... event mask: 000000070000000f [ 0.054286] rcu: Hierarchical SRCU implementation. [ 0.056530] smp: Bringing up secondary CPUs ... [ 0.057521] x86: Booting SMP configuration: [ 0.058033] .... node #0, CPUs: #1 #2 #3 [ 0.062021] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.223073] node 0 deferred pages initialised in 154ms [ 0.226230] devtmpfs: initialized [ 0.227400] x86/mm: Memory block size: 128MB [ 0.232143] gcov: version magic: 0x41383552 [ 0.235344] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.241459] pinctrl core: initialized pinctrl subsystem [ 0.242156] [ 0.242639] ************************************************************* [ 0.245017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247014] ** ** [ 0.249015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251015] ** ** [ 0.253016] ** This means that this kernel is built to expose internal ** [ 0.255016] ** IOMMU data structures, which may compromise security on ** [ 0.257016] ** your system. ** [ 0.259013] ** ** [ 0.261013] ** If you see this message and you are not debugging the ** [ 0.263015] ** kernel, report this immediately to your vendor! ** [ 0.264012] ** ** [ 0.266030] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268017] ************************************************************* [ 0.270480] NET: Registered protocol family 16 [ 0.272543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282073] cpuidle: using governor menu [ 0.283795] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.285626] PCI: Using configuration type 1 for base access [ 0.288158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.296353] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.303157] cryptd: max_cpu_qlen set to 1000 [ 0.306317] ACPI: Added _OSI(Module Device) [ 0.308017] ACPI: Added _OSI(Processor Device) [ 0.310016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.312019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316368] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.321457] ACPI: Interpreter enabled [ 0.322056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323012] ACPI: Using IOAPIC for interrupt routing [ 0.324104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.328430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.336579] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.346080] acpiphp: Slot [2] registered [ 0.348126] acpiphp: Slot [5] registered [ 0.349132] acpiphp: Slot [6] registered [ 0.350133] acpiphp: Slot [7] registered [ 0.351166] acpiphp: Slot [8] registered [ 0.352163] acpiphp: Slot [9] registered [ 0.353104] acpiphp: Slot [10] registered [ 0.354108] acpiphp: Slot [3] registered [ 0.355065] acpiphp: Slot [4] registered [ 0.356030] acpiphp: Slot [11] registered [ 0.357092] acpiphp: Slot [12] registered [ 0.358029] acpiphp: Slot [13] registered [ 0.359117] acpiphp: Slot [14] registered [ 0.360058] acpiphp: Slot [15] registered [ 0.361072] acpiphp: Slot [16] registered [ 0.362028] acpiphp: Slot [17] registered [ 0.362919] acpiphp: Slot [18] registered [ 0.364086] acpiphp: Slot [19] registered [ 0.365116] acpiphp: Slot [20] registered [ 0.366090] acpiphp: Slot [21] registered [ 0.368114] acpiphp: Slot [22] registered [ 0.369130] acpiphp: Slot [23] registered [ 0.370153] acpiphp: Slot [24] registered [ 0.372109] acpiphp: Slot [25] registered [ 0.373136] acpiphp: Slot [26] registered [ 0.374148] acpiphp: Slot [27] registered [ 0.376140] acpiphp: Slot [28] registered [ 0.377109] acpiphp: Slot [29] registered [ 0.379106] acpiphp: Slot [30] registered [ 0.380133] acpiphp: Slot [31] registered [ 0.382071] PCI host bridge to bus 0000:00 [ 0.383042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389089] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.391022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.394030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.396198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.398824] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.401146] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.409726] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.414621] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.417033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.425135] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.427801] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.430051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.432720] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.436942] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.447018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.452009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456790] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.468038] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510715] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.519027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.542023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.552674] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.577018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.588460] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.595017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.602021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.629693] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.639023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.649019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.681296] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.698025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.720049] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.724492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.729387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.731289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.736195] iommu: Default domain type: Passthrough [ 0.738541] SCSI subsystem initialized [ 0.740169] ACPI: bus type USB registered [ 0.741147] usbcore: registered new interface driver usbfs [ 0.743110] usbcore: registered new interface driver hub [ 0.745101] usbcore: registered new device driver usb [ 0.747234] pps_core: LinuxPPS API ver. 1 registered [ 0.749016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752070] PTP clock support registered [ 0.754044] EDAC MC: Ver: 3.0.0 [ 0.756112] PCI: Using ACPI for IRQ routing [ 0.757554] NetLabel: Initializing [ 0.759014] NetLabel: domain hash size = 128 [ 0.760010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761069] NetLabel: unlabeled traffic allowed by default [ 0.763048] vgaarb: loaded [ 0.764214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.771100] clocksource: Switched to clocksource kvm-clock [ 0.893111] VFS: Disk quotas dquot_6.6.0 [ 0.894706] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.897086] *** VALIDATE ramfs *** [ 0.898330] *** VALIDATE hugetlbfs *** [ 0.899811] pnp: PnP ACPI init [ 0.902826] pnp: PnP ACPI: found 6 devices [ 0.932208] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.935364] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937171] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939864] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.942434] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945314] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.949115] NET: Registered protocol family 2 [ 0.952405] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958064] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.962231] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.969682] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.972242] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974677] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977330] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.979521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.981724] NET: Registered protocol family 1 [ 0.983969] RPC: Registered named UNIX socket transport module. [ 0.985201] RPC: Registered udp transport module. [ 0.986145] RPC: Registered tcp transport module. [ 0.987206] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989514] NET: Registered protocol family 44 [ 0.991541] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.993423] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994898] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.996362] PCI: CLS 0 bytes, default 64 [ 0.998734] Unpacking initramfs... [ 2.417869] debug: unmapping init [mem 0xffff8ae4fcc54000-0xffff8ae4fffbffff] [ 2.421577] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.423640] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.426297] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.922259] Initialise system trusted keyrings [ 2.924107] Key type blacklist registered [ 2.926489] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941871] zbud: loaded [ 2.945648] *** VALIDATE nfs *** [ 2.946889] *** VALIDATE nfs4 *** [ 2.948446] pstore: using deflate compression [ 2.952666] Platform Keyring initialized [ 3.081842] NET: Registered protocol family 38 [ 3.083432] Key type asymmetric registered [ 3.084805] Asymmetric key parser 'x509' registered [ 3.086148] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.088900] io scheduler mq-deadline registered [ 3.090059] io scheduler kyber registered [ 3.091242] io scheduler bfq registered [ 3.092435] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.094513] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.096375] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.098411] ACPI: Power Button [PWRF] [ 3.104562] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.109991] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.122052] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.129556] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.144236] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.169141] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196361] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201189] Non-volatile memory driver v1.3 [ 3.202507] Linux agpgart interface v0.103 [ 3.248119] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.251174] vda: detected capacity change from 0 to 75071488 [ 3.273084] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.277591] vdb: detected capacity change from 0 to 1073741824 [ 3.304144] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310642] vdc: detected capacity change from 0 to 2621440000 [ 3.331629] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.334670] vdd: detected capacity change from 0 to 2621440000 [ 3.356967] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.359965] vde: detected capacity change from 0 to 4294967296 [ 3.384480] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389527] vdf: detected capacity change from 0 to 4294967296 [ 3.400911] libphy: Fixed MDIO Bus: probed [ 3.414773] usbcore: registered new interface driver usbserial_generic [ 3.416574] usbserial: USB Serial support registered for generic [ 3.418550] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.423559] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.424908] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.426632] mousedev: PS/2 mouse device common for all mice [ 3.431282] rtc_cmos 00:05: RTC can wake from S4 [ 3.435311] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.438036] rtc_cmos 00:05: registered as rtc0 [ 3.441677] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.444470] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.445245] intel_pstate: CPU model not supported [ 3.450883] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.457114] hid: raw HID events driver (C) Jiri Kosina [ 3.459208] usbcore: registered new interface driver usbhid [ 3.461200] usbhid: USB HID core driver [ 3.462784] drop_monitor: Initializing network drop monitor service [ 3.465156] Initializing XFRM netlink socket [ 3.467296] NET: Registered protocol family 10 [ 3.471178] Segment Routing with IPv6 [ 3.472781] NET: Registered protocol family 17 [ 3.475451] mpls_gso: MPLS GSO support [ 3.481870] RAS: Correctable Errors collector initialized. [ 3.483976] AVX version of gcm_enc/dec engaged. [ 3.485622] AES CTR mode by8 optimization enabled [ 3.574575] sched_clock: Marking stable (3574546923, 0)->(4506197639, -931650716) [ 3.578607] registered taskstats version 1 [ 3.580906] Loading compiled-in X.509 certificates [ 3.583315] zswap: loaded using pool lzo/zbud [ 3.608805] Key type big_key registered [ 3.620167] Key type encrypted registered [ 3.621124] ima: No TPM chip found, activating TPM-bypass! [ 3.622236] ima: Allocated hash algorithm: sha1 [ 3.623774] ima: No architecture policies found [ 3.624830] evm: Initialising EVM extended attributes: [ 3.625933] evm: security.selinux [ 3.626577] evm: security.ima [ 3.627224] evm: security.capability [ 3.627933] evm: HMAC attrs: 0x1 [ 3.630397] rtc_cmos 00:05: setting system clock to 2026-08-26 22:01:23 UTC (1787781683) [ 3.635820] debug: unmapping init [mem 0xffffffffab203000-0xffffffffab3fffff] [ 3.638824] debug: unmapping init [mem 0xffffffffa9f82000-0xffffffffaa258fff] [ 3.648108] Write protecting the kernel read-only data: 28672k [ 3.650759] debug: unmapping init [mem 0xffffffffa8603000-0xffffffffa87fffff] [ 3.653317] debug: unmapping init [mem 0xffffffffa8f14000-0xffffffffa8ffffff] [ 3.690282] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.698689] systemd[1]: Detected virtualization kvm. [ 3.700422] systemd[1]: Detected architecture x86-64. [ 3.702217] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.731857] systemd[1]: No hostname configured. [ 3.733470] systemd[1]: Set hostname to . [ 3.735371] random: systemd: uninitialized urandom read (16 bytes read) [ 3.737801] systemd[1]: Initializing machine ID from random generator. [ 3.789726] random: ln: uninitialized urandom read (6 bytes read) [ 3.869325] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871558] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.875391] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.877959] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] [ 4.048585] random: fast init done Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.487170] device-mapper: uevent: version 1.0.3 [ 4.489864] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.232440] virtio_net virtio0 ens2: renamed from eth0 [ 5.284834] scsi host0: ata_piix [ 5.288055] scsi host1: ata_piix [ 5.290687] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.293143] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.045592] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.133186] random: crng init done [ 10.134865] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.480843] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.518453] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.748500] SELinux: Disabled at runtime. [ 11.808756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.816175] systemd[1]: Detected virtualization kvm. [ 11.817943] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.232336] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.236415] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.241310] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.245952] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.249626] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.257706] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.262472] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 12.303160] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.678762] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.980168] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.020538] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.125461] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.135519] EDAC sbridge: Ver: 1.1.2 [ 14.481304] Key type dns_resolver registered [ 14.785488] NFS: Registering the id_resolver key type [ 14.787696] Key type id_resolver registered [ 14.789279] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg146-server login: [ 41.145284] libcfs: loading out-of-tree module taints kernel. [ 41.167709] Key type ._llcrypt registered [ 41.169496] Key type .llcrypt registered [ 41.226194] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 61.323857] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 63.251349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.282416] alg: No test for adler32 (adler32-zlib) [ 64.822925] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 65.546462] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 67.368805] Key type lgssc registered [ 70.085410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.617541] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.304843] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 134.636093] hrtimer: interrupt took 10205851 ns [ 148.554865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.598893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.124985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.185400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.385373] Lustre: lustre-MDT0000: new disk, initializing [ 150.565364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.591445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.766942] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.858632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.990778] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.028773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.037225] Lustre: Skipped 1 previous similar message [ 174.152551] Lustre: lustre-MDT0001: new disk, initializing [ 174.243306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.269790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.301198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.242935] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.067837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.425885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.804088] Lustre: lustre-OST0000: new disk, initializing [ 195.814954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.824777] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.920149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.640626] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.839707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.864829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.970305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.701786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.854950] Lustre: lustre-OST0001: new disk, initializing [ 220.858644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.864959] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.960663] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.992503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.015492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.073058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.147491] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.610880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.323469] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.352610] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.060597] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing yml_node [ 276.822530] Lustre: DEBUG MARKER: Client: 2.17.57.56 [ 280.196490] Lustre: DEBUG MARKER: MDS: 2.17.57.56 [ 283.247188] Lustre: DEBUG MARKER: OSS: 2.17.57.56 [ 284.920446] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:06:02 EDT 2026 [ 301.313743] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 303.095260] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.040123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.046482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.055719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.553428] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.582671] Lustre: Skipped 1 previous similar message [ 323.598859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.601022] Lustre: Skipped 1 previous similar message [ 325.686606] Lustre: server umount lustre-MDT0000 complete [ 329.564477] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782009 with bad export cookie 6771040434803332707 [ 329.565334] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.580902] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.797778] LustreError: 10514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.804829] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.805530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.824512] LustreError: 10514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 333.836270] Lustre: Skipped 2 previous similar messages [ 334.817595] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.836578] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 336.062918] Lustre: server umount lustre-MDT0001 complete [ 340.902314] Lustre: server umount lustre-OST0000 complete [ 345.264774] Lustre: server umount lustre-OST0001 complete [ 352.880844] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 364.301499] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 403.104059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.379618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.420924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.518922] Lustre: lustre-MDT0000: new disk, initializing [ 403.689590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.742964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.398619] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 418.905959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.044644] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.098976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.102644] Lustre: Skipped 1 previous similar message [ 419.158935] Lustre: lustre-MDT0001: new disk, initializing [ 419.236282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 419.279509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 419.299873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 423.743818] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 434.727227] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.471672] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.362039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 452.849460] Lustre: lustre-OST0000: new disk, initializing [ 452.853996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 452.858611] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.980446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.009274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 454.014598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 454.181127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.143661] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 473.524769] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 479.152433] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 479.390952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.038900] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 485.239466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 489.956214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.959611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.967137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.967147] Lustre: Skipped 3 previous similar messages [ 489.980920] Lustre: Skipped 1 previous similar message [ 489.987518] LustreError: Skipped 1 previous similar message [ 493.707741] Lustre: server umount lustre-OST0000 complete [ 500.202319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.225877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.243130] Lustre: Skipped 2 previous similar messages [ 505.312575] LustreError: 15955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.324492] LustreError: 15955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 506.619836] Lustre: server umount lustre-MDT0000 complete [ 509.410986] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.448780] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 510.828144] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782190 with bad export cookie 6771040434803334240 [ 510.833113] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.844857] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 511.761314] Lustre: server umount lustre-MDT0001 complete [ 516.592448] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:09:53 (1787782193) [ 519.699484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 521.523085] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:09:58 (1787782198) [ 528.819133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.426612] LustreError: 19457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.536866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.215295] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 540.133679] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.169982] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 544.213697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.655771] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 559.438338] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 566.078229] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.509434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 581.883438] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 590.358942] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 599.524791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.547358] Lustre: Skipped 1 previous similar message [ 599.559105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 599.573044] Lustre: Skipped 1 previous similar message [ 604.499261] Lustre: server umount lustre-OST0000 complete [ 614.695438] Lustre: server umount lustre-MDT0000 complete [ 614.880668] LustreError: 21899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.898365] LustreError: 21899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 618.179600] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782298 with bad export cookie 6771040434803335122 [ 618.191242] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.194490] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 618.453459] Lustre: server umount lustre-MDT0001 complete [ 630.985347] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 633.696819] Key type lgssc unregistered [ 634.177429] LNet: 22993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.192697] LNetError: 22993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.233822] LNet: Removed LNI 192.168.201.146@tcp [ 635.222811] Key type .llcrypt unregistered [ 635.227784] Key type ._llcrypt unregistered [ 656.895643] Key type ._llcrypt registered [ 656.899992] Key type .llcrypt registered [ 657.013814] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 658.039276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 658.077715] alg: No test for adler32 (adler32-zlib) [ 659.312779] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 659.781611] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 661.535184] Key type lgssc registered [ 662.892483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 672.681272] loop: module loaded [ 672.818182] loop0: detected capacity change from 0 to 204800000 [ 673.301576] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.346674] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.754062] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.866832] loop0: detected capacity change from 0 to 204800000 [ 682.093306] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.107856] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.123946] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.391213] loop0: detected capacity change from 0 to 204800000 [ 693.564857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.587576] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.923870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 694.993141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 695.135325] Lustre: lustre-MDT0000: new disk, initializing [ 695.354347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.384080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 700.909670] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 710.909158] loop1: detected capacity change from 0 to 204800000 [ 716.537417] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.934490] Lustre: lustre-OST07c6: new disk, initializing [ 716.937600] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 716.940138] Lustre: Skipped 1 previous similar message [ 716.943564] Lustre: 26182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 717.019204] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 723.617679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 723.640563] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 723.675352] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 723.780425] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 729.694499] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 729.701311] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 729.707042] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 729.717083] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 729.731217] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 729.740291] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 754.822975] Lustre: 26801:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 754.846048] Lustre: 26801:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.861142] Lustre: 26801:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.871357] Lustre: 26801:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 754.883238] Lustre: 26801:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 754.900251] Lustre: 26801:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.224728] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 760.235054] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 760.242965] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.258261] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.271663] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.280958] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.285162] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 760.290479] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.295352] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.298655] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.303829] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.307942] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.568738] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 763.579593] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.587792] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 763.604884] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.617514] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.627133] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.633628] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 763.649265] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.655715] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.671211] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.676720] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.683693] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.629255] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 768.648792] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.661320] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.673284] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.691782] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.703606] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.716636] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 768.731771] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.743477] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 768.755805] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.764889] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.772529] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.049618] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 777.057470] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 777.072207] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.083541] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.097897] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.111405] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.117474] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.142372] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.152974] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 777.165832] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.176899] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.186948] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 795.415475] Lustre: 26801:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 795.425553] Lustre: 26801:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 795.432889] Lustre: 26801:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 795.443352] Lustre: 26801:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 795.452147] Lustre: 26801:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 795.461382] Lustre: 26801:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 795.473027] Lustre: 26801:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 795.479781] Lustre: 26801:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 795.491305] Lustre: 26801:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 795.510953] Lustre: 26801:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 795.524122] Lustre: 26801:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 795.530247] Lustre: 26801:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 836.078883] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.100303] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.189654] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 846.303178] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 846.318520] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 846.477433] Lustre: server umount lustre-OST07c6 complete [ 856.799776] Lustre: server umount lustre-MDT0000 complete [ 875.737906] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 878.740280] Key type lgssc unregistered [ 879.035678] LNet: 28748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.044267] LNetError: 28748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.057804] LNet: Removed LNI 192.168.201.146@tcp [ 879.983375] Key type .llcrypt unregistered [ 879.987081] Key type ._llcrypt unregistered [ 897.514567] Key type ._llcrypt registered [ 897.517428] Key type .llcrypt registered [ 897.592903] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 898.665821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.848424] alg: No test for adler32 (adler32-zlib) [ 899.951587] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 900.179530] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 901.847377] Key type lgssc registered [ 903.122848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.272455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.299974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.683602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.050487] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 922.088089] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.114470] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 926.749427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.211545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 931.855932] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 939.811739] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.809519] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 951.131927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.578227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.731640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 957.652188] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 965.791380] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 982.521523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 982.522753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.536535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.547253] Lustre: Skipped 1 previous similar message [ 982.565160] Lustre: Skipped 1 previous similar message [ 987.619261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.230797] Lustre: server umount lustre-OST0000 complete [ 992.738953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.745827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.748517] Lustre: Skipped 1 previous similar message [ 992.774892] Lustre: Skipped 2 previous similar messages [ 997.861912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.869180] Lustre: Skipped 1 previous similar message [ 998.489549] Lustre: server umount lustre-MDT0000 complete [ 1002.571818] LustreError: 30001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782682 with bad export cookie 5799812778467488508 [ 1002.572285] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.580683] LustreError: 30001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1002.959134] Lustre: server umount lustre-MDT0001 complete [ 1014.073015] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1017.954521] Key type lgssc unregistered [ 1018.371583] LNet: 33762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.382961] LNetError: 33762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.408158] LNet: Removed LNI 192.168.201.146@tcp [ 1019.225223] Key type .llcrypt unregistered [ 1019.227910] Key type ._llcrypt unregistered [ 1044.125481] Key type ._llcrypt registered [ 1044.129773] Key type .llcrypt registered [ 1044.270929] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1045.622425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.785724] alg: No test for adler32 (adler32-zlib) [ 1047.019198] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1047.483975] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1049.288443] Key type lgssc registered [ 1050.608582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.429462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.458219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.585538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.074859] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1073.120436] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.177061] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1078.244188] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.093089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.712828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.771465] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1093.876375] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.187542] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.910174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.354935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.012833] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1124.715492] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.949188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1149.411097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.417298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.449353] Lustre: Skipped 1 previous similar message [ 1149.468854] Lustre: Skipped 1 previous similar message [ 1151.681800] Lustre: server umount lustre-OST0000 complete [ 1156.579653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.588126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.608420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.650292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.653180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.690034] Lustre: server umount lustre-MDT0000 complete [ 1164.782432] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.811595] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1166.997839] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782846 with bad export cookie 10067342010193755689 [ 1167.009189] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.012170] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1167.462675] Lustre: server umount lustre-MDT0001 complete [ 1179.571884] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1182.525412] Key type lgssc unregistered [ 1182.812241] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.830085] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.873780] LNet: Removed LNI 192.168.201.146@tcp [ 1183.623209] Key type .llcrypt unregistered [ 1183.624853] Key type ._llcrypt unregistered [ 1205.353147] Key type ._llcrypt registered [ 1205.355219] Key type .llcrypt registered [ 1205.446577] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1206.417210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.438969] alg: No test for adler32 (adler32-zlib) [ 1207.483493] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1207.691569] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1209.343224] Key type lgssc registered [ 1210.800951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1218.410575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1218.440323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.872234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.818160] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1229.284907] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.309597] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1234.403158] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.772441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.574580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.076339] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1250.743911] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1256.723394] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1263.342834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.646340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.714418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1270.010792] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1277.844456] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1287.651189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1287.662141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.676222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.152514] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1292.258508] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.263849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.305778] Lustre: Skipped 2 previous similar messages [ 1293.602071] Lustre: server umount lustre-MDT0000 complete [ 1295.242441] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.384483] LustreError: 40733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.411136] LustreError: 40733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1298.082401] LustreError: 39986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787782977 with bad export cookie 4805374615243275788 [ 1298.096543] LustreError: 39986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1298.097602] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.386537] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [ 1302.497290] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.497887] LustreError: 41740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.509184] Lustre: Skipped 1 previous similar message [ 1302.544852] LustreError: 41740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1304.626418] Lustre: server umount lustre-MDT0001 complete [ 1319.094313] Lustre: server umount lustre-OST0000 complete [ 1335.453828] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1339.055409] Key type lgssc unregistered [ 1339.452659] LNet: 43872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.467611] LNetError: 43872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.506562] LNet: Removed LNI 192.168.201.146@tcp [ 1340.399838] Key type .llcrypt unregistered [ 1340.401905] Key type ._llcrypt unregistered [ 1364.060543] Key type ._llcrypt registered [ 1364.064053] Key type .llcrypt registered [ 1364.157662] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1365.264541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1365.386965] alg: No test for adler32 (adler32-zlib) [ 1366.590839] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1366.964267] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1368.680372] Key type lgssc registered [ 1370.285115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.077576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1379.095479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.663458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.639490] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1391.073661] LustreError: 45290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.092124] LustreError: 45290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1394.098266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.511399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.504026] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1407.847108] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1414.318768] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.325535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.797127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.927588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1429.314232] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1437.800455] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1445.809714] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1446.368585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.373379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.400808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.442994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.453634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.460341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.570234] Lustre: server umount lustre-OST0000 complete [ 1469.594155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.612953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.623137] Lustre: Skipped 2 previous similar messages [ 1475.999183] Lustre: server umount lustre-MDT0000 complete [ 1476.071615] LustreError: 47867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.881829] LustreError: 45269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783159 with bad export cookie 17382873728499962433 [ 1479.893139] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.203773] Lustre: server umount lustre-MDT0001 complete [ 1498.068844] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1500.676880] Key type lgssc unregistered [ 1500.941532] LNet: 49001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.947761] LNetError: 49001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.971733] LNet: Removed LNI 192.168.201.146@tcp [ 1501.851928] Key type .llcrypt unregistered [ 1501.854299] Key type ._llcrypt unregistered [ 1529.514185] Key type ._llcrypt registered [ 1529.520208] Key type .llcrypt registered [ 1529.680272] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1531.263781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.352490] alg: No test for adler32 (adler32-zlib) [ 1532.613174] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1532.936844] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1534.652737] Key type lgssc registered [ 1535.762011] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.263894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.288624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.334732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.814495] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1556.458144] LustreError: 50420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.499551] LustreError: 50420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1561.572265] LustreError: 50419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.370934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.797294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.382617] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1580.403966] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1587.215727] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1595.576452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.158712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.421094] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1611.716696] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1616.697311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1622.231290] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1625.628905] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1637.796903] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2618h28m34s [ 1639.279713] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1642.466477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.470557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.481709] Lustre: Skipped 1 previous similar message [ 1642.493471] Lustre: Skipped 2 previous similar messages [ 1643.571791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.584119] Lustre: Skipped 1 previous similar message [ 1645.568791] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 1645.579081] Lustre: Skipped 1 previous similar message [ 1647.503536] Lustre: server umount lustre-MDT0000 complete [ 1649.635185] LustreError: 50854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.656165] LustreError: 50854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1651.235281] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1655.777171] LustreError: 50416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.804349] LustreError: 50416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1657.747919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.897515] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.273893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.317620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1659.315325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1662.732654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1666.502181] Lustre: DEBUG MARKER: Wait for df (36727) ... [ 1668.760324] Lustre: DEBUG MARKER: done [ 1674.724739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.731342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.750355] Lustre: Skipped 2 previous similar messages [ 1674.763928] Lustre: Skipped 1 previous similar message [ 1679.856437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1679.867412] Lustre: Skipped 1 previous similar message [ 1680.890735] Lustre: server umount lustre-OST0000 complete [ 1690.080927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.086153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1690.097285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.107356] Lustre: Skipped 1 previous similar message [ 1692.055913] Lustre: server umount lustre-MDT0000 complete [ 1696.532848] LustreError: 54594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783376 with bad export cookie 10736988889271988188 [ 1696.534664] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1696.545827] LustreError: 54594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1697.031169] Lustre: server umount lustre-MDT0001 complete [ 1711.320749] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1715.694624] Key type lgssc unregistered [ 1716.108276] LNet: 55350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.118837] LNetError: 55350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.145193] LNet: Removed LNI 192.168.201.146@tcp [ 1717.349347] Key type .llcrypt unregistered [ 1717.351144] Key type ._llcrypt unregistered [ 1757.888862] Key type ._llcrypt registered [ 1757.893601] Key type .llcrypt registered [ 1758.045709] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1759.170279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.199747] alg: No test for adler32 (adler32-zlib) [ 1760.472718] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1760.805675] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1762.471478] Key type lgssc registered [ 1763.460824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.306855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.336109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.698314] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1773.744478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1774.006600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.764041] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1789.237705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.384791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1789.440217] Lustre: 57558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1789.737300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1795.508771] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1803.716092] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.768145] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1820.144667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.486049] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1820.851896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.010197] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 1827.428660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1836.146512] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.186295] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2618h32m01s [ 1845.961670] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1853.837104] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1856.961530] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1857.056933] LustreError: 57568:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1877.415468] Lustre: lustre-MDT0000: Client 71796356-096c-4532-af8c-21111c2d312b (at 192.168.201.46@tcp) reconnecting [ 1897.127220] LustreError: 57568:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1899.580913] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1901.929332] Lustre: DEBUG MARKER: done [ 1908.916389] Lustre: server umount lustre-OST0000 complete [ 1914.849718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.859521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.866393] Lustre: Skipped 1 previous similar message [ 1915.942722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.956739] Lustre: Skipped 1 previous similar message [ 1922.018972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.024150] Lustre: Skipped 1 previous similar message [ 1923.133652] Lustre: server umount lustre-MDT0000 complete [ 1926.117892] Lustre: 56884:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783589/real 1787783589] req@00000000d91b6ea7 x1874626808438400/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787783605 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1926.154129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.179680] Lustre: Skipped 1 previous similar message [ 1927.027067] LustreError: 57551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783606 with bad export cookie 9504647344441407269 [ 1927.042841] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.348957] Lustre: server umount lustre-MDT0001 complete [ 1939.531883] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 1943.084814] Key type lgssc unregistered [ 1943.562538] LNet: 61893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.572139] LNetError: 61893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.593891] LNet: Removed LNI 192.168.201.146@tcp [ 1944.880376] Key type .llcrypt unregistered [ 1944.883632] Key type ._llcrypt unregistered [ 1991.907187] Key type ._llcrypt registered [ 1991.908764] Key type .llcrypt registered [ 1992.025702] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 1993.045716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.162940] alg: No test for adler32 (adler32-zlib) [ 1994.312700] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 1994.699889] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 1996.410152] Key type lgssc registered [ 1997.509713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.528558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.547930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.821122] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2001.834458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2001.997170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.905835] Lustre: Failing over lustre-MDT0000 [ 2003.199248] Lustre: server umount lustre-MDT0000 complete [ 2010.022104] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:34:47 (1787783687) [ 2016.460093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.937187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.783851] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2031.937620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.111636] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2032.183652] Lustre: 64694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2032.514861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.112283] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2044.118825] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2049.241689] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.752715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.986757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2057.221842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.466365] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2065.516221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2071.098058] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.341658] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2083.808518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.817462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.849640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.369614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.373603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.384667] Lustre: Skipped 1 previous similar message [ 2088.993160] Lustre: server umount lustre-MDT0000 complete [ 2092.431706] LustreError: 64686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783772 with bad export cookie 12318856738051583343 [ 2092.436097] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.444036] LustreError: 64686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2092.838531] Lustre: server umount lustre-MDT0001 complete [ 2096.721835] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2101.112865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2109.458791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.992232] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783776/real 1787783776] req@00000000a5ce6f8d x1874627053789312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2113.016376] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.029355] Lustre: Skipped 1 previous similar message [ 2114.015876] Lustre: 63618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783777/real 1787783777] req@000000003392b482 x1874627053789440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.466022] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.486788] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2117.552839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.624956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2118.559151] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783782/real 1787783782] req@000000009b9adff9 x1874627053789824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2118.575302] LustreError: 66544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.600431] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2122.003790] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2122.600258] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.639219] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783787/real 1787783787] req@0000000096e4ae4a x1874627053789952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.735190] Lustre: 63619:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783792/real 1787783792] req@000000002047f3f3 x1874627053790080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.864522] LustreError: 68464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.878249] LustreError: 68464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2130.171176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.591505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.951343] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2136.058214] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2142.256308] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2142.780175] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.066142] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.016767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.034923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.048550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.060621] Lustre: Skipped 1 previous similar message [ 2161.820634] Lustre: server umount lustre-MDT0000 complete [ 2162.663440] LustreError: 70089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.677812] LustreError: 70089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2165.258557] LustreError: 66551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783845 with bad export cookie 12318856738051585618 [ 2165.262372] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.269468] LustreError: 66551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2165.633574] Lustre: server umount lustre-MDT0001 complete [ 2171.571687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2180.235960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.135514] Lustre: 63617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787783847/real 1787783847] req@0000000049e343e7 x1874627053817984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787783863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.171882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.196322] Lustre: Skipped 2 previous similar messages [ 2190.368383] LustreError: 63615:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009aa6c1fb x1874627053819008/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2190.746141] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.818074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.869051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2195.187697] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2204.149805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.563640] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2209.775064] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2212.839553] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2216.416684] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.935765] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2228.195705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.198660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.214355] Lustre: Skipped 2 previous similar messages [ 2228.226392] Lustre: Skipped 5 previous similar messages [ 2229.984172] Lustre: server umount lustre-MDT0000 complete [ 2233.315462] LustreError: 71872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.329988] LustreError: 71872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2233.658689] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783913 with bad export cookie 12318856738051586997 [ 2233.669539] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.674262] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2233.905959] Lustre: server umount lustre-MDT0001 complete [ 2239.603540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2243.179402] Lustre: DEBUG MARKER: files compared the same [ 2255.665741] Lustre: server umount lustre-OST0000 complete [ 2276.134206] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2279.160493] Key type lgssc unregistered [ 2279.383511] LNet: 74424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.394426] LNetError: 74424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.409514] LNet: Removed LNI 192.168.201.146@tcp [ 2280.107041] Key type .llcrypt unregistered [ 2280.111456] Key type ._llcrypt unregistered [ 2309.035515] Key type ._llcrypt registered [ 2309.038785] Key type .llcrypt registered [ 2309.196562] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2310.546604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.596342] alg: No test for adler32 (adler32-zlib) [ 2312.005401] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2312.393326] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2314.191167] Key type lgssc registered [ 2315.579330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.343393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.389821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.059184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.668400] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2335.200408] LustreError: 75841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.210691] LustreError: 75841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2340.249062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.328023] LustreError: 75842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.920416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.286322] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2353.577964] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2358.526966] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2365.333166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2365.473059] LustreError: 77664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.545246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.677605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2371.803621] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug +malloc all [ 2380.036188] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2391.520023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2391.526176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.532796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.547826] Lustre: Skipped 1 previous similar message [ 2391.563410] Lustre: Skipped 1 previous similar message [ 2395.370483] Lustre: server umount lustre-OST0000 complete [ 2401.759945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.762961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.778695] Lustre: Skipped 1 previous similar message [ 2401.793801] Lustre: Skipped 1 previous similar message [ 2405.630882] Lustre: server umount lustre-MDT0000 complete [ 2406.881287] LustreError: 76785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.909748] LustreError: 76785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2409.511738] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784089 with bad export cookie 18116908624727647779 [ 2409.515463] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.524852] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2410.169985] Lustre: server umount lustre-MDT0001 complete [ 2421.507451] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2424.302197] Key type lgssc unregistered [ 2424.667946] LNet: 79380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.683425] LNetError: 79380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.698549] LNet: Removed LNI 192.168.201.146@tcp [ 2425.808132] Key type .llcrypt unregistered [ 2425.818277] Key type ._llcrypt unregistered [ 2454.136408] Key type ._llcrypt registered [ 2454.138861] Key type .llcrypt registered [ 2454.288915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2455.893302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.194831] alg: No test for adler32 (adler32-zlib) [ 2457.364656] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2457.698547] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2459.456060] Key type lgssc registered [ 2460.773728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2468.659213] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2468.680630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.256704] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2489.376539] LustreError: 80790:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [ 2516.063810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.329733] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2531.646429] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.039644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.192758] Lustre: *** cfs_fail_loc=706, val=0*** [ 2539.542951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.602470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2544.323441] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2544.608832] LustreError: 80816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.619397] LustreError: 80816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2549.736434] LustreError: 80815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.768274] LustreError: 81875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.787783] LustreError: 81875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2552.848138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.198953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.200770] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2564.079590] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.863554] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2573.806386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.816308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2573.830082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.849104] Lustre: server umount lustre-OST0000 complete [ 2584.037749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.044663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2584.049808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.049817] Lustre: Skipped 1 previous similar message [ 2584.059834] Lustre: Skipped 2 previous similar messages [ 2589.036911] Lustre: server umount lustre-MDT0000 complete [ 2589.153350] LustreError: 83449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.176807] LustreError: 83449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2593.262434] LustreError: 83499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.307614] LustreError: 83499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2593.364709] LustreError: 80797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784273 with bad export cookie 10541880090724034446 [ 2593.372797] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.380411] LustreError: 80797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2593.998844] Lustre: server umount lustre-MDT0001 complete [ 2605.562568] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 2608.253250] Key type lgssc unregistered [ 2608.575910] LNet: 84408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2608.593301] LNetError: 84408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2608.617954] LNet: Removed LNI 192.168.201.146@tcp [ 2609.721144] Key type .llcrypt unregistered [ 2609.723375] Key type ._llcrypt unregistered [ 2638.882273] Key type ._llcrypt registered [ 2638.884425] Key type .llcrypt registered [ 2638.989671] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2640.022037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2640.224081] alg: No test for adler32 (adler32-zlib) [ 2641.536954] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 2642.010913] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 2643.831561] Key type lgssc registered [ 2645.072577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2652.912708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2652.926488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.434221] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2666.889419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.526805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2672.467822] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2672.609443] LustreError: 86512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.635059] LustreError: 86512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2677.730172] LustreError: 86516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.784343] LustreError: 86516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2681.961144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.262203] LustreError: 86512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.278627] LustreError: 86512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2682.293497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.460498] LustreError: 86511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.500912] LustreError: 86511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2689.476439] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2692.581774] LustreError: 86515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.631073] LustreError: 86515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2699.728734] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.743875] Lustre: MGS: Client 2e67a04c-6ac2-42f4-b9d8-112143af0a95 (at 0@lo) reconnecting [ 2699.778425] LustreError: 86526:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2700.169112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.272705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2706.709785] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2720.740014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.751980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2720.763945] Lustre: Skipped 1 previous similar message [ 2720.764067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2725.875288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2725.884253] Lustre: Skipped 2 previous similar messages [ 2726.482625] Lustre: server umount lustre-OST0000 complete [ 2730.988764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.992801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.003335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.016531] Lustre: Skipped 1 previous similar message [ 2731.024343] Lustre: Skipped 1 previous similar message [ 2736.097227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.103895] Lustre: Skipped 1 previous similar message [ 2736.471432] Lustre: server umount lustre-MDT0000 complete [ 2740.618570] Lustre: server umount lustre-MDT0001 complete [ 2744.295145] Lustre: server umount MGS complete [ 2755.323569] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:47:12 (1787784432) [ 2787.432457] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 2797.028382] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 2841.975791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.800874] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2862.038985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.368454] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2862.422751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2862.526372] Lustre: lustre-MDT0001: new disk, initializing [ 2862.637457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.670722] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2862.678773] Lustre: 93687:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2862.683121] LustreError: 93687:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2867.343943] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2880.682452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.911180] Lustre: lustre-OST0000: new disk, initializing [ 2880.914740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2880.922188] Lustre: 94454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2881.001142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.985602] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2890.336440] Lustre: 94878:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002a8577b1 x1874627732217728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787784586 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2899.770533] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.654123] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2907.672686] Lustre: MGS: Client 2d978812-bd97-4fe3-8432-c616cf676610 (at 0@lo) reconnecting [ 2907.753811] Lustre: 92956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2907.771683] LustreError: 93673:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2907.786547] LustreError: 93673:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2907.819390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2907.902571] Lustre: lustre-MDT0000: new disk, initializing [ 2908.074931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.104826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2908.140054] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2909.223483] Lustre: 95523:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000090b908a9 x1874627732225024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787784605 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2914.019633] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 2914.401194] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2917.896912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2917.905074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2918.467389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2943.468669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.475515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.497083] Lustre: Skipped 1 previous similar message [ 2943.497466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.526888] Lustre: Skipped 1 previous similar message [ 2946.894731] Lustre: server umount lustre-OST0000 complete [ 2953.697724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2953.699203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.699883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.747433] Lustre: Skipped 1 previous similar message [ 2957.928220] Lustre: server umount lustre-MDT0000 complete [ 2958.817630] LustreError: 93659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.838714] LustreError: 93659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2968.906233] Lustre: server umount lustre-MDT0001 complete [ 2974.048778] Lustre: server umount MGS complete [ 2989.137233] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:51:05 (1787784665) [ 3009.509067] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3012.075788] Key type lgssc unregistered [ 3012.333741] LNet: 98027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.338839] LNetError: 98027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.354852] LNet: Removed LNI 192.168.201.146@tcp [ 3013.247561] Key type .llcrypt unregistered [ 3013.250072] Key type ._llcrypt unregistered [ 3032.496350] Key type ._llcrypt registered [ 3032.505599] Key type .llcrypt registered [ 3032.685357] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3033.880257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.164710] alg: No test for adler32 (adler32-zlib) [ 3035.179798] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3035.398422] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3037.063165] Key type lgssc registered [ 3038.624972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.450868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.490606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.281045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.925303] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3060.725612] LustreError: 99248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.775970] LustreError: 99248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3061.731383] LustreError: 99249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.856495] LustreError: 99248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.071865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.464050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3071.529841] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3080.940443] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3087.890119] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3096.793403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.001474] LustreError: 101069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.312593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.709763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3104.281941] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3113.454729] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.173876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3123.174303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.187500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.211933] Lustre: Skipped 1 previous similar message [ 3127.689381] Lustre: server umount lustre-OST0000 complete [ 3133.408595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.437745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.444431] Lustre: Skipped 2 previous similar messages [ 3138.528840] LustreError: 99248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.541623] LustreError: 99248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3140.558322] Lustre: server umount lustre-MDT0000 complete [ 3144.603160] LustreError: 99228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784824 with bad export cookie 14363679538443979470 [ 3144.611737] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.622833] LustreError: 99228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3145.211272] Lustre: server umount lustre-MDT0001 complete [ 3158.625915] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3162.051904] Key type lgssc unregistered [ 3162.451385] LNet: 102785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.487943] LNetError: 102785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.561768] LNet: Removed LNI 192.168.201.146@tcp [ 3164.661841] Key type .llcrypt unregistered [ 3164.664652] Key type ._llcrypt unregistered [ 3184.003708] Key type ._llcrypt registered [ 3184.005341] Key type .llcrypt registered [ 3184.156071] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3186.303187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3186.502677] alg: No test for adler32 (adler32-zlib) [ 3187.462811] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3187.688806] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3189.367382] Key type lgssc registered [ 3190.958225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.491536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.501461] LustreError: 103860:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3197.529430] LustreError: 103869:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3197.537021] LustreError: 103860:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3197.545126] LustreError: 103860:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3197.556357] LustreError: 103860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3197.576747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.146250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.900940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.248774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.690531] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3219.597966] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing lsmod [ 3223.225247] LustreError: 105074:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3223.257826] LustreError: 105078:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3223.268629] LustreError: 105074:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 3223.287246] LustreError: 105074:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3223.319469] LustreError: 105074:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3223.332229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.039689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.617155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.631659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.665891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.157370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.163193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.172054] Lustre: Skipped 1 previous similar message [ 3232.295664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.312712] Lustre: Skipped 4 previous similar messages [ 3233.789452] Lustre: server umount lustre-MDT0000 complete [ 3237.349790] LustreError: 103906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.362705] LustreError: 103906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3237.561447] LustreError: 105116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784917 with bad export cookie 2655039086195556745 [ 3237.572372] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.574743] LustreError: 105116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3238.495326] LustreError: 104338:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007f9dffa4 x1874628304722048/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.952942] Lustre: server umount lustre-MDT0001 complete [ 3252.567395] Lustre: server umount lustre-OST0000 complete [ 3260.666089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.180380] LustreError: 106141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.303688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.539351] LustreError: 106142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.632820] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3271.658959] LustreError: 106141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.583884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.010266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.414600] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3288.270539] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3295.167258] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3302.589403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.949497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.066580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3309.540447] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3317.946351] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3332.583864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3332.599193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.611687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.660692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.673143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.205254] Lustre: server umount lustre-OST0000 complete [ 3345.892184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.903847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.921344] Lustre: Skipped 1 previous similar message [ 3345.940991] Lustre: Skipped 3 previous similar messages [ 3349.321682] Lustre: server umount lustre-MDT0000 complete [ 3351.008655] LustreError: 106138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.027975] LustreError: 106138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3353.184145] LustreError: 106123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785033 with bad export cookie 2655039086195557445 [ 3353.195246] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.506589] Lustre: server umount lustre-MDT0001 complete [ 3364.430870] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3367.004309] Key type lgssc unregistered [ 3367.315761] LNet: 109701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.325806] LNetError: 109701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.344759] LNet: Removed LNI 192.168.201.146@tcp [ 3368.446944] Key type .llcrypt unregistered [ 3368.448810] Key type ._llcrypt unregistered [ 3399.789563] Key type ._llcrypt registered [ 3399.798084] Key type .llcrypt registered [ 3399.952537] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3402.145408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3402.229070] alg: No test for adler32 (adler32-zlib) [ 3403.678935] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3404.375149] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3406.183372] Key type lgssc registered [ 3408.313748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.188554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3416.209678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.901637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.318526] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3427.297740] LustreError: 111122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.305634] LustreError: 111122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3432.421421] LustreError: 111123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.830574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3434.459151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3439.717296] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3449.399715] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3456.046930] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3464.356957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.854124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.516953] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3480.386852] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3487.433144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3496.177386] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3500.336896] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3508.198329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.199772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.204239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.210825] Lustre: Skipped 1 previous similar message [ 3511.430766] Lustre: server umount lustre-OST0000 complete [ 3516.390106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3516.397707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.418922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.421770] Lustre: Skipped 1 previous similar message [ 3518.435200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.443091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.162498] Lustre: server umount lustre-MDT0000 complete [ 3523.558974] LustreError: 111118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.579228] LustreError: 111118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3526.225911] LustreError: 113724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785206 with bad export cookie 17384588283669054990 [ 3526.230808] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3526.234808] LustreError: 113724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3526.728071] Lustre: server umount lustre-MDT0001 complete [ 3539.242110] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3542.300639] Key type lgssc unregistered [ 3542.716763] LNet: 114784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3542.728284] LNetError: 114784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3542.774445] LNet: Removed LNI 192.168.201.146@tcp [ 3543.810771] Key type .llcrypt unregistered [ 3543.819464] Key type ._llcrypt unregistered [ 3563.932677] Key type ._llcrypt registered [ 3563.934971] Key type .llcrypt registered [ 3564.032284] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3565.504461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3565.598232] alg: No test for adler32 (adler32-zlib) [ 3566.784312] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3567.073823] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3568.904172] Key type lgssc registered [ 3570.349808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3579.499433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3579.542191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.062032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.844689] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3590.625393] LustreError: 116008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.659946] LustreError: 116008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3595.747596] LustreError: 116007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.752066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.231541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3600.830490] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3610.593532] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.117213] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3624.496539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.883622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.922900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3632.549431] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3640.020558] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3650.539573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.548719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.562651] Lustre: Skipped 1 previous similar message [ 3650.564667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.589834] Lustre: Skipped 1 previous similar message [ 3655.351433] Lustre: server umount lustre-OST0000 complete [ 3660.770307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.788070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.173837] Lustre: server umount lustre-MDT0000 complete [ 3665.894332] LustreError: 116007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.920633] LustreError: 116007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3668.847181] LustreError: 115990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785348 with bad export cookie 17947003744785634956 [ 3668.856688] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.860477] LustreError: 115990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3669.290726] Lustre: server umount lustre-MDT0001 complete [ 3680.229580] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3683.231759] Key type lgssc unregistered [ 3683.688438] LNet: 119559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3683.696804] LNetError: 119559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3683.716086] LNet: Removed LNI 192.168.201.146@tcp [ 3684.562432] Key type .llcrypt unregistered [ 3684.570857] Key type ._llcrypt unregistered [ 3710.401684] Key type ._llcrypt registered [ 3710.411295] Key type .llcrypt registered [ 3710.558230] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3712.054066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3712.266096] alg: No test for adler32 (adler32-zlib) [ 3713.301951] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3713.569517] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3715.215329] Key type lgssc registered [ 3716.615015] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.703560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3725.721900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.225699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.942931] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3737.577445] LustreError: 120979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.598836] LustreError: 120979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3740.820165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.179944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3746.351219] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3753.140815] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.660320] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.262155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.526599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.928273] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3772.792334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3779.294323] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3788.398578] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3794.053796] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3794.065550] Lustre: Skipped 2 previous similar messages [ 3794.523161] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3794.559956] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3798.979469] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3798.991744] Lustre: Skipped 2 previous similar messages [ 3802.822236] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3802.831653] Lustre: Skipped 1 previous similar message [ 3808.564650] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3808.573389] Lustre: Skipped 2 previous similar messages [ 3809.154292] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.46@tcp 192.168.0.[2,10]@tcp [ 3815.524993] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.46/24@tcp in log lustre-MDT0000 [ 3815.539740] Lustre: Skipped 2 previous similar messages [ 3824.752864] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.46/24@tcp [ 3824.762692] Lustre: Skipped 1 previous similar message [ 3833.329329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3833.336436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.350402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.344866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.349660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3834.365909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.634564] Lustre: server umount lustre-OST0000 complete [ 3843.554641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3843.569825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.591295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.601762] Lustre: Skipped 1 previous similar message [ 3847.851429] Lustre: server umount lustre-MDT0000 complete [ 3849.699617] LustreError: 120974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.721917] LustreError: 120974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3852.248274] LustreError: 120959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785532 with bad export cookie 5796436162423494735 [ 3852.248543] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.267027] LustreError: 120959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3852.686901] Lustre: server umount lustre-MDT0001 complete [ 3865.127376] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 3868.951726] Key type lgssc unregistered [ 3869.417442] LNet: 125831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3869.428835] LNetError: 125831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3869.448060] LNet: Removed LNI 192.168.201.146@tcp [ 3870.530833] Key type .llcrypt unregistered [ 3870.534310] Key type ._llcrypt unregistered [ 3906.853635] Key type ._llcrypt registered [ 3906.855908] Key type .llcrypt registered [ 3906.924064] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 3908.016638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3908.170856] alg: No test for adler32 (adler32-zlib) [ 3909.513361] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 3909.652774] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 3911.255476] Key type lgssc registered [ 3912.096349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3919.828260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3919.857978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.519880] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3921.543300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.587549] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.46/24@tcp [ 3926.736530] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3932.135845] LustreError: 127396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.149489] LustreError: 127396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3937.283646] LustreError: 127395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.908333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.577148] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3940.625765] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.46/24@tcp [ 3940.715944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3945.177836] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3953.689640] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3959.557403] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3968.250870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.640256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.173560] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 3984.261681] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3992.193758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4006.894868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4006.912403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.953843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.916064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4007.916612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.919599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.284475] Lustre: server umount lustre-OST0000 complete [ 4017.636483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.645936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.689756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.702355] Lustre: Skipped 1 previous similar message [ 4022.512156] Lustre: server umount lustre-MDT0000 complete [ 4023.270539] LustreError: 128413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.307665] LustreError: 128413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4026.599280] LustreError: 127375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785706 with bad export cookie 6712449744072633580 [ 4026.609871] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.998252] Lustre: server umount lustre-MDT0001 complete [ 4039.720736] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4042.359656] Key type lgssc unregistered [ 4042.666557] LNet: 131017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4042.671308] LNetError: 131017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4042.689062] LNet: Removed LNI 192.168.201.146@tcp [ 4043.650767] Key type .llcrypt unregistered [ 4043.652462] Key type ._llcrypt unregistered [ 4071.381232] Key type ._llcrypt registered [ 4071.383680] Key type .llcrypt registered [ 4071.557908] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4090.818770] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4091.819898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.902280] alg: No test for adler32 (adler32-zlib) [ 4092.998206] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 4093.187866] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4094.815100] Key type lgssc registered [ 4095.926022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4131.268500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4131.305610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.523687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4132.556953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4132.628974] Lustre: lustre-MDT0000: new disk, initializing [ 4132.716581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.734891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4137.618151] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4147.942657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.092352] Lustre: 134424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4148.143365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4148.146482] Lustre: Skipped 1 previous similar message [ 4148.311650] Lustre: lustre-MDT0001: new disk, initializing [ 4148.478381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4148.547622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4148.572521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4152.905393] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4161.950257] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4167.892271] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.711483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.053414] Lustre: lustre-OST0000: new disk, initializing [ 4175.061394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4175.070357] Lustre: 136389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4175.163728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.857628] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4183.638891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4183.654875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4183.788859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4192.222313] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4198.477208] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4198.771665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4203.624375] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4203.951704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4209.666821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4209.667617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.670175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4209.674802] LustreError: Skipped 1 previous similar message [ 4209.725291] Lustre: Skipped 1 previous similar message [ 4214.772698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.791577] Lustre: Skipped 2 previous similar messages [ 4219.876595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.881744] Lustre: Skipped 1 previous similar message [ 4220.383214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.580541] Lustre: server umount lustre-OST0000 complete [ 4225.003082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.005283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.013452] Lustre: Skipped 1 previous similar message [ 4225.031640] Lustre: Skipped 1 previous similar message [ 4230.114872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.119719] Lustre: Skipped 1 previous similar message [ 4231.080501] Lustre: server umount lustre-MDT0000 complete [ 4235.347073] LustreError: 134415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785915 with bad export cookie 6625518232177254572 [ 4235.360170] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.895403] Lustre: server umount lustre-MDT0001 complete [ 4246.624616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.241570] LustreError: 138609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.275139] LustreError: 138609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4247.364259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.648745] LustreError: 138610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.606655] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4257.782088] LustreError: 138609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.890622] LustreError: 138610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.360584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.763204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4270.139947] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4278.515775] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4284.457557] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4291.438242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4292.047146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.450601] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4307.629748] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.088937] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4313.335474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4319.249624] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4319.495945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4329.892612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4330.129637] Lustre: lustre-OST0001: new disk, initializing [ 4330.134530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4330.140988] Lustre: 141981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4330.243116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4337.862123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4337.883716] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4338.061491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4338.471648] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4348.017567] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4348.318467] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4352.927578] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4353.244276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4358.985369] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ad0c6a0e000.ost_server_uuid 50 [ 4360.540358] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ad0c6a0e000.ost_server_uuid in FULL state after 0 sec [ 4371.643972] Lustre: server umount lustre-OST0001 complete [ 4373.996151] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4374.001298] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.029034] LustreError: 143285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.037523] Lustre: Skipped 1 previous similar message [ 4374.062208] LustreError: 143285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4376.034509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4376.034773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.045390] LustreError: Skipped 2 previous similar messages [ 4376.053594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.227519] LustreError: 140434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.262197] LustreError: 140434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4389.855221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4390.323774] Lustre: server umount lustre-OST0000 complete [ 4398.051360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4398.061124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.076920] Lustre: Skipped 1 previous similar message [ 4398.081238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4398.088290] Lustre: Skipped 7 previous similar messages [ 4403.165233] Lustre: server umount lustre-MDT0000 complete [ 4404.704881] LustreError: 138610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.716228] LustreError: 138610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4407.058952] LustreError: 138590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786086 with bad export cookie 6625518232177255454 [ 4407.062231] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.066772] LustreError: 138590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4407.559360] Lustre: server umount lustre-MDT0001 complete [ 4429.115101] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4432.744801] Key type lgssc unregistered [ 4433.133465] LNet: 144991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4433.145322] LNetError: 144991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4433.161528] LNet: Removed LNI 192.168.201.146@tcp [ 4434.300194] Key type .llcrypt unregistered [ 4434.302672] Key type ._llcrypt unregistered [ 4479.218191] Key type ._llcrypt registered [ 4479.219610] Key type .llcrypt registered [ 4479.342573] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4497.114303] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4498.560660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4498.857849] alg: No test for adler32 (adler32-zlib) [ 4499.915055] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 4500.325652] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4502.103202] Key type lgssc registered [ 4503.634417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4542.984366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4543.007435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.456512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4544.500296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4544.662368] Lustre: lustre-MDT0000: new disk, initializing [ 4544.823214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.942192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4549.939466] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4563.306295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.521974] Lustre: 149151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4563.641353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4563.651672] Lustre: Skipped 1 previous similar message [ 4563.836664] Lustre: lustre-MDT0001: new disk, initializing [ 4564.014821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4564.081607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4564.108972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4570.443165] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4580.187305] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4585.886433] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4592.429695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4592.679332] Lustre: lustre-OST0000: new disk, initializing [ 4592.682514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4592.686802] Lustre: 151119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4592.739932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.593351] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4602.445247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4602.454765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4602.609435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4607.568189] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4611.371668] Lustre: Failing over lustre-OST0000 [ 4611.498748] Lustre: server umount lustre-OST0000 complete [ 4628.431111] Lustre: 147375:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786292/real 1787786292] req@00000000d21270c8 x1874629681513088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787786308 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4628.444791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.456229] Lustre: Skipped 1 previous similar message [ 4629.473223] Lustre: 147376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786292/real 1787786292] req@000000009f54c1bb x1874629681512064/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787786308 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4629.506529] Lustre: 147376:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4629.534759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4630.041648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.078065] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4634.597950] Lustre: 147376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786298/real 1787786298] req@00000000c8cd1261 x1874629681514112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787786314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.013731] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4636.111795] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4639.179645] Lustre: 147374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786303/real 1787786303] req@00000000aedac1f8 x1874629681515264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787786319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4639.216510] Lustre: 147374:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4639.265489] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4639.270137] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4639.985337] Lustre: Failing over lustre-MDT0000 [ 4640.749051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.766071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.255994] Lustre: server umount lustre-MDT0000 complete [ 4644.335328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.349847] Lustre: Skipped 1 previous similar message [ 4645.254051] LustreError: 149157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.283512] LustreError: 149157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4649.442394] LustreError: 149158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.471384] LustreError: 149158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4654.562469] LustreError: 149163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.590944] LustreError: 149163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4659.688106] LustreError: 149157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.714306] LustreError: 149157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4660.707685] Lustre: 147374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786324/real 1787786324] req@00000000420435e9 x1874629681525376/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787786340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4660.747178] Lustre: 147374:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4660.766890] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4661.897617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.479641] LustreError: 149158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.523201] LustreError: 149158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4670.961101] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7dfa445eecacf58d [ 4670.987083] Lustre: MGC192.168.201.146@tcp: Connection restored to 0@lo (at 0@lo) [ 4670.989925] Lustre: Skipped 1 previous similar message [ 4671.538903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.636598] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4671.671551] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4676.620182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4676.683345] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4676.782345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4677.497329] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4686.825821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.830780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.846023] Lustre: Skipped 1 previous similar message [ 4691.937607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.951902] Lustre: Skipped 2 previous similar messages [ 4692.988101] Lustre: server umount lustre-OST0000 complete [ 4697.058741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.063263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.081019] Lustre: Skipped 1 previous similar message [ 4697.136712] Lustre: Skipped 1 previous similar message [ 4702.187995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4702.198168] Lustre: Skipped 1 previous similar message [ 4702.995141] Lustre: server umount lustre-MDT0000 complete [ 4706.686726] LustreError: 152914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786386 with bad export cookie 9077643173407028621 [ 4706.688586] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4706.700158] LustreError: 152914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4706.920488] Lustre: server umount lustre-MDT0001 complete [ 4718.838896] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 4721.592902] Key type lgssc unregistered [ 4721.918985] LNet: 154777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4721.933575] LNetError: 154777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4721.951354] LNet: Removed LNI 192.168.201.146@tcp [ 4722.828111] Key type .llcrypt unregistered [ 4722.830486] Key type ._llcrypt unregistered [ 4762.238603] Key type ._llcrypt registered [ 4762.241382] Key type .llcrypt registered [ 4762.428221] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4763.602360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4763.640063] alg: No test for adler32 (adler32-zlib) [ 4764.780562] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 4765.057730] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 4766.927274] Key type lgssc registered [ 4768.076457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4775.854733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4775.877252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.264204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.661633] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4787.695425] LustreError: 156301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.733588] LustreError: 156301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4791.476839] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4792.804044] LustreError: 156302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.375511] LustreError: 156330:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4795.391833] LustreError: 156330:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4795.397234] LustreError: 156330:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4795.763607] Lustre: server umount lustre-MDT0000 complete [ 4810.191872] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 4820.628769] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 4857.499344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.769843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4857.803638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4857.878386] Lustre: lustre-MDT0000: new disk, initializing [ 4857.947707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.964924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4862.257418] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4872.498951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.628204] Lustre: 159637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4872.682922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4872.687631] Lustre: Skipped 1 previous similar message [ 4872.737174] Lustre: lustre-MDT0001: new disk, initializing [ 4872.808203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4872.847962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4872.859816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4877.344564] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4889.199216] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4895.470638] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4904.554867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.882721] Lustre: lustre-OST0000: new disk, initializing [ 4904.891455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4904.904655] Lustre: 161606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4905.057188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.760234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4911.777760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4912.013840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4913.012592] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 4924.145488] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4929.478450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4930.948504] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:23:28 (1787786608) [ 5144.033462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5144.054628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.073251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.617951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.622873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.635208] Lustre: Skipped 1 previous similar message [ 5147.647704] Lustre: Skipped 1 previous similar message [ 5152.736903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.743779] Lustre: Skipped 3 previous similar messages [ 5157.343220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5157.743493] Lustre: server umount lustre-MDT0000 complete [ 5157.871152] LustreError: 159648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.892990] LustreError: 159648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5161.547456] LustreError: 159627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786841 with bad export cookie 17530899021800280643 [ 5161.556728] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.566707] LustreError: 159627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5162.976484] LustreError: 160743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.981873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.026515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5167.021800] LustreError: 159644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.163544] Lustre: server umount lustre-MDT0001 complete [ 5178.964290] Lustre: server umount lustre-OST0000 complete [ 5190.831856] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5204.277619] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5245.913570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.205150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5246.251559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5246.390733] Lustre: lustre-MDT0000: new disk, initializing [ 5246.477956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.493613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5250.792296] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5261.673645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.854479] Lustre: 166265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5261.920634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5261.928676] Lustre: Skipped 1 previous similar message [ 5261.990780] Lustre: lustre-MDT0001: new disk, initializing [ 5262.057277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5262.092909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5262.114418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5267.132969] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5276.775744] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5282.219558] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5288.925635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5289.249337] Lustre: lustre-OST0000: new disk, initializing [ 5289.256461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5289.270472] Lustre: 168232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5289.431617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.208041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5295.219272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5295.349929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5296.091075] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5306.500995] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5312.520785] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5312.730504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5318.527909] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5318.796957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5326.320635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5326.335850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.348204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5326.348213] Lustre: Skipped 1 previous similar message [ 5326.405245] Lustre: Skipped 1 previous similar message [ 5327.630429] Lustre: server umount lustre-OST0000 complete [ 5334.016367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5334.042282] LustreError: Skipped 1 previous similar message [ 5334.056942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.550336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5336.567499] Lustre: Skipped 3 previous similar messages [ 5338.879810] Lustre: server umount lustre-MDT0000 complete [ 5341.681931] LustreError: 166277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.704368] LustreError: 166277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5343.506748] LustreError: 166256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787023 with bad export cookie 17530899021800439473 [ 5343.514125] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5343.516961] LustreError: 166256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5343.824869] Lustre: server umount lustre-MDT0001 complete [ 5353.008279] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:30:30 (1787787030) [ 5370.617771] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5381.403716] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5422.346860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.495383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5422.513898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5422.572719] Lustre: lustre-MDT0000: new disk, initializing [ 5422.619112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.633705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5427.582784] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5439.614640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.733463] Lustre: 172989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5440.017171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5444.844327] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5456.485865] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5463.849381] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5471.446455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5471.831171] Lustre: lustre-OST0000: new disk, initializing [ 5471.834809] Lustre: Skipped 1 previous similar message [ 5471.837962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5471.841170] Lustre: Skipped 2 previous similar messages [ 5471.847656] Lustre: 174972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5478.833332] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5478.969430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5478.980979] Lustre: Skipped 1 previous similar message [ 5478.987308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5479.188191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5488.810904] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.875043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5499.892703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.896016] LustreError: Skipped 1 previous similar message [ 5499.917515] Lustre: Skipped 1 previous similar message [ 5499.918409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5499.939819] Lustre: Skipped 1 previous similar message [ 5502.576396] Lustre: server umount lustre-OST0000 complete [ 5506.528435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5512.849978] Lustre: server umount lustre-MDT0000 complete [ 5515.233610] LustreError: 174139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.246821] LustreError: 174139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5516.388501] LustreError: 172983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787196 with bad export cookie 17530899021800440355 [ 5516.397380] LustreError: 172983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5516.403701] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.685549] Lustre: server umount lustre-MDT0001 complete [ 5526.017634] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:33:23 (1787787203) [ 5544.043186] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 5554.204061] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5596.440714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.718803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5596.746541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5596.841219] Lustre: lustre-MDT0000: new disk, initializing [ 5596.947718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.950583] Lustre: Skipped 2 previous similar messages [ 5596.968242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5602.569312] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5613.762951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.928548] Lustre: 179296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5614.282653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5618.962784] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5628.802058] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.520619] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5642.358624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5642.585563] Lustre: 181272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5649.830947] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5652.546424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5652.718505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5659.930265] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.321388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.324310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.326325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.326330] Lustre: Skipped 3 previous similar messages [ 5668.339594] Lustre: Skipped 4 previous similar messages [ 5673.461040] LustreError: 181251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.620605] Lustre: server umount lustre-OST0000 complete [ 5692.383147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5692.638262] Lustre: server umount lustre-MDT0000 complete [ 5696.343508] LustreError: 179289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787376 with bad export cookie 17530899021800441440 [ 5696.352098] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.818920] Lustre: server umount lustre-MDT0001 complete [ 5710.739371] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5714.106390] Key type lgssc unregistered [ 5714.477668] LNet: 183137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5714.484552] LNetError: 183137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5714.507398] LNet: Removed LNI 192.168.201.146@tcp [ 5715.529421] Key type .llcrypt unregistered [ 5715.534275] Key type ._llcrypt unregistered [ 5742.702714] Key type ._llcrypt registered [ 5742.706731] Key type .llcrypt registered [ 5742.837905] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5744.672374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5744.756926] alg: No test for adler32 (adler32-zlib) [ 5745.815135] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5746.131685] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5747.815204] Key type lgssc registered [ 5749.298150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.201762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5758.231055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.855684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.548577] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5769.191296] LustreError: 184556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.220565] LustreError: 184556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5774.310584] LustreError: 184555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.632340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.982908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5780.185583] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5788.314556] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5795.294222] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5802.302284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5802.539844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.605935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5809.174903] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5817.613776] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.314314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.318556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.333328] Lustre: Skipped 1 previous similar message [ 5838.353511] Lustre: Skipped 1 previous similar message [ 5842.713936] Lustre: server umount lustre-OST0000 complete [ 5848.545759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.552836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.562427] Lustre: Skipped 1 previous similar message [ 5848.587799] Lustre: Skipped 1 previous similar message [ 5853.670268] LustreError: 184555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.705488] LustreError: 184555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5853.905834] Lustre: server umount lustre-MDT0000 complete [ 5857.937839] LustreError: 186398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787537 with bad export cookie 16582124882447067118 [ 5857.948604] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.958265] LustreError: 186398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5858.342460] Lustre: server umount lustre-MDT0001 complete [ 5870.874384] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 5875.046381] Key type lgssc unregistered [ 5875.561587] LNet: 188114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5875.573721] LNetError: 188114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5875.611285] LNet: Removed LNI 192.168.201.146@tcp [ 5877.271205] Key type .llcrypt unregistered [ 5877.278929] Key type ._llcrypt unregistered [ 5908.516627] Key type ._llcrypt registered [ 5908.519472] Key type .llcrypt registered [ 5908.632508] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 5910.294465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.557548] alg: No test for adler32 (adler32-zlib) [ 5911.735324] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 5912.055558] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 5913.743864] Key type lgssc registered [ 5915.492820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5924.954829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5924.982686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.710126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.352897] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5937.123800] LustreError: 189534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.142068] LustreError: 189534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5941.148647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.598935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5947.282780] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5955.996909] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5962.320496] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5970.575814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5971.033235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.122894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5978.252359] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 5986.831684] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5994.917042] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 5997.032835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5997.034368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.051335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5997.062715] Lustre: Skipped 1 previous similar message [ 5997.086954] Lustre: Skipped 1 previous similar message [ 6000.001302] Lustre: lustre-OST0000: Not available for connect from 192.168.201.46@tcp (stopping) [ 6000.725420] Lustre: server umount lustre-OST0000 complete [ 6009.660781] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ad0c3639800.ost_server_uuid 50 [ 6011.540830] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6031.254566] Lustre: server umount lustre-MDT0000 complete [ 6032.870779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.882480] LustreError: 189531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.893261] LustreError: 189531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6035.445581] LustreError: 189514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787715 with bad export cookie 12245885727870711997 [ 6035.489130] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6036.107392] Lustre: server umount lustre-MDT0001 complete [ 6047.216081] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:42:03 (1787787723) [ 6055.444212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.827118] LustreError: 193452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.886771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.390716] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6061.025261] LustreError: 193453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.146471] LustreError: 193452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.070620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.700394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.969140] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6083.103490] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.604636] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6098.446523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.742072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.836013] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6105.776171] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6114.618297] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6121.896968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6122.181689] Lustre: lustre-OST0001: new disk, initializing [ 6122.187910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6122.194522] Lustre: 196291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6122.288939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.248493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6128.261865] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6128.388594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6130.148848] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6140.760455] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6144.992562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.005762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.048471] Lustre: Skipped 1 previous similar message [ 6145.067036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.064615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.083673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.101074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.110475] Lustre: Skipped 3 previous similar messages [ 6150.508240] Lustre: server umount lustre-OST0000 complete [ 6152.076971] LustreError: 196582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.097643] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6158.284594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6163.071834] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6163.212615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6169.583502] LustreError: 196582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.630766] LustreError: 196582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6179.808372] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.813547] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.826497] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6179.879233] LustreError: Skipped 1 previous similar message [ 6181.665312] Lustre: server umount lustre-OST0001 complete [ 6187.490568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6190.053465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.080679] Lustre: Skipped 2 previous similar messages [ 6193.137214] Lustre: server umount lustre-MDT0000 complete [ 6198.075796] LustreError: 193434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787877 with bad export cookie 12245885727870713068 [ 6198.078681] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6198.094117] LustreError: 193434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6199.014320] Lustre: server umount lustre-MDT0001 complete [ 6227.148371] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:45:04 (1787787904) [ 6234.765995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.036129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6235.086212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.414673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.474426] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6250.774808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.883054] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6250.940451] Lustre: 199733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6256.985654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6267.841277] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6273.909420] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6281.148319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6281.430671] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6288.347649] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6291.203881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6297.510923] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6306.023231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.139378] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6306.564373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6306.567550] Lustre: Skipped 2 previous similar messages [ 6313.108098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6313.640741] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6324.022541] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6327.272859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.273567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.284359] LustreError: Skipped 1 previous similar message [ 6327.298277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.313506] Lustre: Skipped 5 previous similar messages [ 6332.873813] Lustre: server umount lustre-OST0000 complete [ 6333.938161] LustreError: 201582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.953586] LustreError: 201582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6353.901509] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6357.422769] Lustre: server umount lustre-OST0001 complete [ 6364.130217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.148473] Lustre: Skipped 3 previous similar messages [ 6364.157653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.161375] Lustre: Skipped 6 previous similar messages [ 6367.991610] Lustre: server umount lustre-MDT0000 complete [ 6372.121282] LustreError: 203800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788051 with bad export cookie 12245885727870714475 [ 6372.131935] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.137878] LustreError: 203800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6372.559120] Lustre: server umount lustre-MDT0001 complete [ 6401.173526] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:47:57 (1787788077) [ 6421.321947] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 6433.462593] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 6471.669743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.931851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6471.954398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6472.039112] Lustre: lustre-MDT0000: new disk, initializing [ 6472.188937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6472.238128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6477.828817] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6491.631632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.758756] Lustre: 207841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.825548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6491.830128] Lustre: Skipped 1 previous similar message [ 6491.945408] Lustre: lustre-MDT0001: new disk, initializing [ 6492.086468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6492.108956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6497.337101] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6508.543602] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6514.438698] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6522.387149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.778911] Lustre: lustre-OST0000: new disk, initializing [ 6522.788451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6522.810684] Lustre: 209809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6528.544073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6528.559362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6528.658738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6529.545847] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6538.389976] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6544.890270] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6545.292904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6551.567626] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6551.966835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6554.597753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6554.636035] Lustre: Skipped 1 previous similar message [ 6554.654946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6554.664897] Lustre: Skipped 1 previous similar message [ 6559.202272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6559.218996] LustreError: Skipped 2 previous similar messages [ 6560.713736] Lustre: server umount lustre-OST0000 complete [ 6579.167318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6579.599410] Lustre: server umount lustre-MDT0000 complete [ 6583.534431] LustreError: 207833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788263 with bad export cookie 12245885727870716260 [ 6583.544201] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6583.554583] LustreError: 207833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6583.914830] Lustre: server umount lustre-MDT0001 complete [ 6594.001246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.538877] LustreError: 212026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.551771] LustreError: 212026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6599.899461] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6609.938655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.902740] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6624.969730] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6632.255557] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6639.942986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.216680] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6655.788816] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6661.572339] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6661.808921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6667.032704] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6667.226815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6671.332371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6675.106199] Lustre: server umount lustre-OST0000 complete [ 6681.968837] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6686.929654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6715.987171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.309372] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6731.094937] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6734.868818] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6736.900913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6736.923591] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6736.926063] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6736.949837] Lustre: Skipped 1 previous similar message [ 6741.996560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.012125] Lustre: Skipped 6 previous similar messages [ 6742.016166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6742.021629] Lustre: Skipped 13 previous similar messages [ 6744.995395] Lustre: server umount lustre-OST0000 complete [ 6755.250220] Lustre: server umount lustre-MDT0000 complete [ 6757.353284] LustreError: 212027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.365817] LustreError: 212027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6758.654446] LustreError: 212006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788438 with bad export cookie 12245885727870717142 [ 6758.675300] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6759.288963] Lustre: server umount lustre-MDT0001 complete [ 6768.719925] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:54:05 (1787788445) [ 6775.964965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.703501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.711779] Lustre: Skipped 6 previous similar messages [ 6781.795244] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6790.348787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.777263] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6804.649295] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6810.216569] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6816.446841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.886400] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6830.464361] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6835.331981] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6835.528913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6840.201552] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6840.377698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6846.978291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6847.245129] Lustre: lustre-OST0001: new disk, initializing [ 6847.252424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6847.261840] Lustre: 221369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6854.085320] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6854.796903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6854.829852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6854.987434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6864.360731] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6869.052267] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6869.273968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6874.673242] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6874.957844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6880.736454] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.739271] LustreError: Skipped 3 previous similar messages [ 6882.874795] Lustre: server umount lustre-OST0001 complete [ 6890.287285] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6890.651057] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6895.394222] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6895.569605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6926.114647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6931.954713] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6931.978945] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6931.983423] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6932.022947] Lustre: Skipped 1 previous similar message [ 6933.947853] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 6941.044718] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.705657] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6959.075458] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6959.155476] Lustre: server umount lustre-OST0001 complete [ 6981.581720] Lustre: server umount lustre-OST0000 complete [ 6991.621651] Lustre: server umount lustre-MDT0000 complete [ 6995.353546] LustreError: 217978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788675 with bad export cookie 12245885727870718108 [ 6995.356389] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.363416] LustreError: 217978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7001.855754] Lustre: server umount lustre-MDT0001 complete [ 7024.454886] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:58:21 (1787788701) [ 7030.987285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.279298] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7031.313258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7036.320735] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7045.902905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.046229] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7046.096224] Lustre: 226614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7050.582832] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7057.544426] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7063.064080] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7070.120327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.482100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7078.845264] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7088.764786] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.287158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.343512] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7105.215212] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7112.587651] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.021303] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7118.642224] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7118.802686] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7123.595064] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7123.798817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7130.024255] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ad0d09aa800.ost_server_uuid 50 [ 7131.573770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ad0d09aa800.ost_server_uuid in IDLE state after 0 sec [ 7132.682477] Lustre: Permanently deactivating lustre-OST0001 [ 7132.690774] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7135.733600] Lustre: Permanently reactivating lustre-OST0001 [ 7141.361821] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.371548] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7141.383482] Lustre: Skipped 11 previous similar messages [ 7141.394580] Lustre: Skipped 15 previous similar messages [ 7141.423683] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7141.439313] LustreError: Skipped 4 previous similar messages [ 7145.333602] Lustre: server umount lustre-OST0001 complete [ 7146.471795] LustreError: 228461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.513165] LustreError: 228461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 7155.931965] Lustre: server umount lustre-OST0000 complete [ 7175.137032] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7175.579357] Lustre: server umount lustre-MDT0000 complete [ 7180.487439] LustreError: 228476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788860 with bad export cookie 12245885727870719396 [ 7180.500884] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7180.503563] LustreError: 228476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7180.873401] Lustre: server umount lustre-MDT0001 complete [ 7207.421916] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:01:24 (1787788884) [ 7215.307702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.537525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7215.567325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7215.574610] Lustre: Skipped 5 previous similar messages [ 7220.094789] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7228.682471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.749589] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7228.767354] Lustre: 233321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7233.988688] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7242.597857] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7248.544337] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7256.194737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.452526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7256.479786] Lustre: Permanently deactivating lustre-OST0000 [ 7256.507669] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7262.744071] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7263.601642] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7272.719796] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7281.353298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7281.447051] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7287.876328] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7291.714870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7295.459811] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7297.471832] Lustre: Permanently reactivating lustre-OST0000 [ 7297.500776] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7297.503853] Lustre: Skipped 2 previous similar messages [ 7299.199581] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7299.203142] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 000000009b5b909d, cur 1787788979 deadline 1787788972 last 1787788942 [ 7299.216477] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7299.217310] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7299.265634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7299.315772] LustreError: 235505:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -107 [ 7300.393748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7327.199358] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7327.342630] Lustre: server umount lustre-OST0001 complete [ 7337.327612] Lustre: server umount lustre-OST0000 complete [ 7355.360851] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7355.538152] Lustre: server umount lustre-MDT0000 complete [ 7359.417084] LustreError: 233313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789039 with bad export cookie 12245885727870721041 [ 7359.422626] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7359.428656] LustreError: 233313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7359.918293] Lustre: server umount lustre-MDT0001 complete [ 7371.821572] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7374.487189] Key type lgssc unregistered [ 7374.771628] LNet: 238210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7374.777794] LNetError: 238210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7374.788646] LNet: Removed LNI 192.168.201.146@tcp [ 7375.741877] Key type .llcrypt unregistered [ 7375.743814] Key type ._llcrypt unregistered [ 7399.892888] Key type ._llcrypt registered [ 7399.895423] Key type .llcrypt registered [ 7399.996650] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7401.078424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7401.152159] alg: No test for adler32 (adler32-zlib) [ 7402.280289] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7402.540130] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7404.191170] Key type lgssc registered [ 7405.165137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7414.332288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7414.358231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.792864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7420.648312] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7426.021446] LustreError: 239674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.053343] LustreError: 239674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7428.879981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.008175] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7429.036736] Lustre: 239664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7429.043868] Lustre: Permanently deactivating lustre-MDT0001 [ 7429.055655] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7429.287253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7433.412758] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7433.431983] LustreError: 239705:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7433.450086] LustreError: 239705:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7433.465980] LustreError: 239705:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7434.205955] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7443.516624] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7448.766819] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7456.006339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7456.316864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.391194] LustreError: 241518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.401806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7462.763388] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7465.442509] LustreError: 241740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.347963] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7470.570702] LustreError: 241518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.666202] LustreError: 241740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.715330] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7477.125831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.414109] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7482.868622] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7482.873439] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7483.391412] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7490.629286] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7493.033943] Lustre: Permanently reactivating lustre-MDT0001 [ 7493.046720] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7493.051957] Lustre: Skipped 2 previous similar messages [ 7496.163309] Lustre: lustre-MDT0001: Client 4710fd42-4c2f-4b06-940d-f3cb0e678e48 (at 192.168.201.46@tcp) reconnecting [ 7501.769535] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7503.468634] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7509.501979] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7509.697386] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7514.386867] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7514.627798] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7517.166760] Lustre: Permanently deactivating lustre-MDT0001 [ 7517.193291] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7517.197309] Lustre: Skipped 2 previous similar messages [ 7524.558335] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7531.491480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7531.503815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.516620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7531.957541] Lustre: server umount lustre-MDT0000 complete [ 7534.053684] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.055411] LustreError: 239674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.066107] Lustre: Skipped 1 previous similar message [ 7534.097972] LustreError: 239674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7535.739434] LustreError: 239655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789215 with bad export cookie 5988387718900820722 [ 7535.748805] LustreError: 239655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7535.758631] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7536.110753] Lustre: server umount lustre-MDT0001 complete [ 7541.857398] Lustre: server umount lustre-OST0000 complete [ 7553.174809] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:07:10 (1787789230) [ 7565.486560] Lustre: server umount lustre-OST0001 complete [ 7574.008223] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 7589.829516] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7624.513388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.871890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7624.888446] Lustre: Skipped 2 previous similar messages [ 7624.911157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7625.000858] Lustre: lustre-MDT0000: new disk, initializing [ 7625.091941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.115421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7629.586987] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7640.287873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.446764] Lustre: 247956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7640.492324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7640.495785] Lustre: Skipped 1 previous similar message [ 7640.575698] Lustre: lustre-MDT0001: new disk, initializing [ 7640.664295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.694579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7640.709535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7646.791232] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7657.892739] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7664.229611] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7671.460495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.755803] Lustre: lustre-OST0000: new disk, initializing [ 7671.759874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7671.779865] Lustre: 249925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7671.881702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.350978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7678.580480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7678.601278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7678.767393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7688.432192] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7690.544693] LustreError: 247964:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7699.466053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.605688] Lustre: lustre-OST0001: new disk, initializing [ 7699.609236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7699.614642] Lustre: 251038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7700.640331] LustreError: 247964:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7700.678968] LustreError: 247963:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7705.171049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7705.189166] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7705.273559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7705.856202] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7710.713643] LustreError: 247963:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7714.925293] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7716.154589] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7720.897087] Lustre: lustre-OST0001: Not available for connect from 192.168.201.46@tcp (stopping) [ 7720.930407] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.946622] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.961553] Lustre: Skipped 2 previous similar messages [ 7724.665359] Lustre: server umount lustre-OST0001 complete [ 7725.964061] LustreError: 249910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.504478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.507845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.520745] LustreError: Skipped 2 previous similar messages [ 7745.524569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.530763] Lustre: Skipped 1 previous similar message [ 7745.553163] Lustre: Skipped 3 previous similar messages [ 7748.997912] Lustre: server umount lustre-OST0000 complete [ 7756.786541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.801957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.813675] Lustre: Skipped 1 previous similar message [ 7756.822064] Lustre: Skipped 3 previous similar messages [ 7759.831997] Lustre: server umount lustre-MDT0000 complete [ 7761.895868] LustreError: 247964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.911928] LustreError: 247964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7764.878390] LustreError: 251037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789444 with bad export cookie 5988387718900823536 [ 7764.884962] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.894754] LustreError: 251037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7779.295944] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7780.022842] Lustre: server umount lustre-MDT0001 complete [ 7792.498959] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 7795.560364] Key type lgssc unregistered [ 7795.922291] LNet: 253076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7795.942592] LNetError: 253076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7795.966822] LNet: Removed LNI 192.168.201.146@tcp [ 7797.490363] Key type .llcrypt unregistered [ 7797.492599] Key type ._llcrypt unregistered [ 7847.874766] Key type ._llcrypt registered [ 7847.876892] Key type .llcrypt registered [ 7847.953319] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 7848.794337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7848.954541] alg: No test for adler32 (adler32-zlib) [ 7849.934403] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 7850.160360] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 7851.855355] Key type lgssc registered [ 7853.086203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.932143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7858.957865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.110772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7860.128477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7860.244962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.772571] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7871.368609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.494639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7871.546306] Lustre: 255286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7871.772275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7875.764534] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7881.834330] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7886.703637] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7893.760600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.876955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7894.064689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.792525] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7903.418789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7906.614508] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.522295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7959.539800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.575687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.038723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7960.047839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.283949] Lustre: server umount lustre-OST0000 complete [ 7966.345722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7978.721240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7979.376447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.391354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7980.411126] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7985.512318] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7985.535783] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7987.193560] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 7996.193095] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.791783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.803299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.808386] Lustre: Skipped 2 previous similar messages [ 8039.906652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8039.919504] Lustre: Skipped 1 previous similar message [ 8041.088678] Lustre: server umount lustre-OST0000 complete [ 8045.023906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.026724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.032144] Lustre: Skipped 2 previous similar messages [ 8050.146453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.155678] Lustre: Skipped 2 previous similar messages [ 8059.359293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8059.790353] Lustre: server umount lustre-MDT0000 complete [ 8063.187865] LustreError: 255277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789743 with bad export cookie 1261641967101751413 [ 8063.189527] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.200555] LustreError: 255277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8063.604487] Lustre: server umount lustre-MDT0001 complete [ 8073.882764] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8076.405754] Key type lgssc unregistered [ 8076.688307] LNet: 260712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8076.699422] LNetError: 260712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8077.735066] LNet: Removed LNI 192.168.201.146@tcp [ 8078.515656] Key type .llcrypt unregistered [ 8078.517497] Key type ._llcrypt unregistered [ 8102.437673] Key type ._llcrypt registered [ 8102.439935] Key type .llcrypt registered [ 8102.594505] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8103.659209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8103.765087] alg: No test for adler32 (adler32-zlib) [ 8104.797558] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8105.066488] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8106.855280] Key type lgssc registered [ 8107.812796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8114.689815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8114.701560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.318496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.295228] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8126.441241] LustreError: 262132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.452077] LustreError: 262132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8128.486770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.765729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8133.194990] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8139.801298] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8144.522549] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.017870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8151.455955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.809344] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8161.716037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8163.918806] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8187.362457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.368765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.373209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.375378] Lustre: Skipped 1 previous similar message [ 8187.393984] Lustre: Skipped 1 previous similar message [ 8191.874643] Lustre: server umount lustre-OST0000 complete [ 8195.552500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.565807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.577989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.601707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.603256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.516750] Lustre: server umount lustre-MDT0000 complete [ 8202.720478] LustreError: 262133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.742594] LustreError: 262133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8204.855215] LustreError: 262112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789884 with bad export cookie 4841151008783338736 [ 8204.856773] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.867212] LustreError: 262112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8205.092201] Lustre: server umount lustre-MDT0001 complete [ 8215.057940] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8217.535953] Key type lgssc unregistered [ 8217.768903] LNet: 266438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8217.777522] LNetError: 266438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8217.788239] LNet: Removed LNI 192.168.201.146@tcp [ 8218.500892] Key type .llcrypt unregistered [ 8218.502808] Key type ._llcrypt unregistered [ 8236.352965] Key type ._llcrypt registered [ 8236.356531] Key type .llcrypt registered [ 8236.442841] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8237.332078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.369702] alg: No test for adler32 (adler32-zlib) [ 8238.325976] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8238.489976] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8240.113180] Key type lgssc registered [ 8240.911716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8247.196516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8247.227885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.831440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8253.668889] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8259.045457] LustreError: 267660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.060952] LustreError: 267660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8261.551791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.920204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.700261] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8272.921291] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.184410] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8285.012041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8285.425245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8292.636512] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8293.686775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8300.423528] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8312.289203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8312.295270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.306962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.337145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.350823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.373194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.863211] Lustre: server umount lustre-OST0000 complete [ 8319.457078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.463522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8319.469568] Lustre: Skipped 2 previous similar messages [ 8324.582611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.591370] Lustre: Skipped 1 previous similar message [ 8325.730042] Lustre: server umount lustre-MDT0000 complete [ 8329.072362] LustreError: 267640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790008 with bad export cookie 4549458669244244954 [ 8329.078894] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.085122] LustreError: 267640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8329.368486] Lustre: server umount lustre-MDT0001 complete [ 8341.252974] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8344.605656] Key type lgssc unregistered [ 8344.852643] LNet: 271340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8344.856596] LNetError: 271340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8345.894631] LNet: Removed LNI 192.168.201.146@tcp [ 8346.885770] Key type .llcrypt unregistered [ 8346.887858] Key type ._llcrypt unregistered [ 8378.066837] Key type ._llcrypt registered [ 8378.075665] Key type .llcrypt registered [ 8378.228483] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8379.429540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8379.563079] alg: No test for adler32 (adler32-zlib) [ 8380.566094] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8380.783695] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8382.431559] Key type lgssc registered [ 8383.604314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8390.349204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8390.362972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.670400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.936188] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8401.894808] LustreError: 272758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.912850] LustreError: 272758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8401.997637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.429944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8406.036692] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8411.913688] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8416.776488] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8422.554569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.861658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.698266] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8434.966960] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8436.208610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8458.731234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8458.741944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.756388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.806761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.809432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8461.813579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.289268] Lustre: server umount lustre-OST0000 complete [ 8466.913067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.925838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.928403] Lustre: Skipped 2 previous similar messages [ 8472.033447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8472.039351] Lustre: Skipped 1 previous similar message [ 8472.268589] Lustre: server umount lustre-MDT0000 complete [ 8475.536572] LustreError: 276108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790155 with bad export cookie 201731963266081841 [ 8475.543987] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8475.546107] LustreError: 276108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8476.041214] Lustre: server umount lustre-MDT0001 complete [ 8484.477318] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8486.688254] Key type lgssc unregistered [ 8486.969783] LNet: 277066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8486.973743] LNetError: 277066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8486.987895] LNet: Removed LNI 192.168.201.146@tcp [ 8487.564967] Key type .llcrypt unregistered [ 8487.566619] Key type ._llcrypt unregistered [ 8504.189975] Key type ._llcrypt registered [ 8504.192114] Key type .llcrypt registered [ 8504.342273] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8505.318738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8505.361388] alg: No test for adler32 (adler32-zlib) [ 8506.453515] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8506.689280] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8508.383181] Key type lgssc registered [ 8509.325733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8516.187855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8516.227414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.804176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.399458] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8527.336138] LustreError: 278306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.356541] LustreError: 278306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8531.350870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.799665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8536.798410] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8545.186828] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8550.120528] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8556.508440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8557.082891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.224966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8562.429729] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8568.542762] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.747546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8587.758399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.775301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.853258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8591.855198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.858136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.849721] Lustre: server umount lustre-OST0000 complete [ 8596.961380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.964424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.972578] Lustre: Skipped 1 previous similar message [ 8596.974709] Lustre: Skipped 1 previous similar message [ 8602.082283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8602.084956] Lustre: Skipped 2 previous similar messages [ 8607.204030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8607.214770] Lustre: Skipped 1 previous similar message [ 8611.296162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8611.613844] Lustre: server umount lustre-MDT0000 complete [ 8614.794916] LustreError: 278271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790294 with bad export cookie 8157437864010630129 [ 8614.808338] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8614.816235] LustreError: 278271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8615.138990] Lustre: server umount lustre-MDT0001 complete [ 8624.131171] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8626.623069] Key type lgssc unregistered [ 8626.849907] LNet: 282030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8626.856210] LNetError: 282030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8626.874580] LNet: Removed LNI 192.168.201.146@tcp [ 8627.571845] Key type .llcrypt unregistered [ 8627.573239] Key type ._llcrypt unregistered [ 8637.149473] Key type ._llcrypt registered [ 8637.152245] Key type .llcrypt registered [ 8637.257745] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8655.654459] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 8669.567607] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8670.953916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8671.003484] alg: No test for adler32 (adler32-zlib) [ 8672.100633] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8672.329727] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8674.015171] Key type lgssc registered [ 8675.074820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8705.347486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8705.382876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.665140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8706.715305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8706.817648] Lustre: lustre-MDT0000: new disk, initializing [ 8706.914039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.928378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8711.038020] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8720.157089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8720.331556] Lustre: 285582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8720.407511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8720.411626] Lustre: Skipped 1 previous similar message [ 8720.489283] Lustre: lustre-MDT0001: new disk, initializing [ 8720.604705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8720.659211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8720.666797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8724.503903] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8732.054488] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8737.187528] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8743.059648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.272305] Lustre: lustre-OST0000: new disk, initializing [ 8743.275940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8743.282718] Lustre: 287552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8743.370507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.710210] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8749.636795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8749.651445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8749.739868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8754.588892] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8758.597626] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8758.762731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8761.974967] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8762.170228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8765.409746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8765.411565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.423244] LustreError: Skipped 1 previous similar message [ 8765.426291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8765.450913] Lustre: Skipped 1 previous similar message [ 8765.473363] Lustre: Skipped 1 previous similar message [ 8770.536154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8770.539442] Lustre: Skipped 1 previous similar message [ 8775.661505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.668536] Lustre: Skipped 1 previous similar message [ 8778.207995] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8778.411727] Lustre: server umount lustre-OST0000 complete [ 8782.304229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8782.307497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.319302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.895026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.298221] Lustre: server umount lustre-MDT0000 complete [ 8792.099800] LustreError: 286379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790471 with bad export cookie 12326625393358554111 [ 8792.101677] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8792.111760] LustreError: 286379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8792.743297] Lustre: server umount lustre-MDT0001 complete [ 8801.949830] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:27:59 (1787790479) [ 8808.118896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8808.408570] LustreError: 289965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.428426] LustreError: 289965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8808.478045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.128065] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8813.536459] LustreError: 289966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.665229] LustreError: 289965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8819.586027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.833845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8823.268683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8829.142951] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8833.527427] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8838.816986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.134096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8844.559642] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 8850.584818] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8864.736911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8864.744587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8864.759402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8864.761888] Lustre: Skipped 2 previous similar messages [ 8868.843984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8868.844756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.958737] Lustre: server umount lustre-OST0000 complete [ 8876.003567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8876.011694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8876.018486] Lustre: Skipped 2 previous similar messages [ 8879.073744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.095669] Lustre: Skipped 1 previous similar message [ 8880.460654] Lustre: server umount lustre-MDT0000 complete [ 8883.560508] LustreError: 289946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790563 with bad export cookie 12326625393358554993 [ 8883.565812] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.569502] LustreError: 289946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8883.866834] Lustre: server umount lustre-MDT0001 complete [ 8893.143896] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 8895.861566] Key type lgssc unregistered [ 8896.090636] LNet: 293522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8896.096386] LNetError: 293522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8896.109897] LNet: Removed LNI 192.168.201.146@tcp [ 8896.842920] Key type .llcrypt unregistered [ 8896.850235] Key type ._llcrypt unregistered [ 8921.369586] Key type ._llcrypt registered [ 8921.376118] Key type .llcrypt registered [ 8921.539714] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 8935.860760] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 8936.978312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8937.054482] alg: No test for adler32 (adler32-zlib) [ 8938.107802] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 8938.329617] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 8939.967182] Key type lgssc registered [ 8941.083267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8985.468810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8985.490995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.726334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.765751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8986.866494] Lustre: lustre-MDT0000: new disk, initializing [ 8986.967701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.993950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8992.221325] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9001.354791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9001.474369] Lustre: 297345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9001.527342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9001.530784] Lustre: Skipped 1 previous similar message [ 9001.599690] Lustre: lustre-MDT0001: new disk, initializing [ 9001.689206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9001.733076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9001.742779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9005.913192] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9014.407294] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9019.186497] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9024.621971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9024.857275] Lustre: lustre-OST2710: new disk, initializing [ 9024.860800] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9024.867292] Lustre: 299314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9024.947815] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9030.249431] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9033.259972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9033.269129] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9033.392075] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9037.970991] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9043.630676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9043.769370] Lustre: lustre-OST03e8: new disk, initializing [ 9043.772739] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9043.778483] Lustre: 300378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9043.863718] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9049.335904] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9053.276647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9053.285732] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9053.369873] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9055.336634] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9062.227320] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9062.420593] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9066.680994] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9066.863239] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9071.337462] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9071.539939] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9075.334279] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9075.518807] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9079.277385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.281340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9079.295688] Lustre: Skipped 2 previous similar messages [ 9079.310878] Lustre: Skipped 3 previous similar messages [ 9081.329646] Lustre: server umount lustre-MDT0000 complete [ 9084.797018] LustreError: 297338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790764 with bad export cookie 7924735861641989769 [ 9084.798414] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.808049] LustreError: 297338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9085.147516] Lustre: server umount lustre-MDT0001 complete [ 9098.804366] Lustre: server umount lustre-OST2710 complete [ 9112.865390] Lustre: server umount lustre-OST03e8 complete [ 9129.795868] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [ 9140.671274] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9173.437467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.669446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9173.702333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9173.767149] Lustre: lustre-MDT0000: new disk, initializing [ 9173.852334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.870606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9177.690487] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9187.374814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.539689] Lustre: 305632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9187.583957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9187.587996] Lustre: Skipped 1 previous similar message [ 9187.642120] Lustre: lustre-MDT0001: new disk, initializing [ 9187.717406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9187.770121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9187.785063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9191.391891] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9200.020773] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9205.561937] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9213.419575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9213.798041] Lustre: lustre-OST0000: new disk, initializing [ 9213.805523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9213.813625] Lustre: 307600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9213.951404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.240483] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9221.753824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9221.771304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9221.920488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9228.808584] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9233.181589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9233.419802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9237.070792] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9237.244775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9242.596158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9242.599463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.608817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9242.610181] LustreError: Skipped 1 previous similar message [ 9242.637588] Lustre: Skipped 2 previous similar messages [ 9247.714585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9247.719986] Lustre: Skipped 2 previous similar messages [ 9252.841737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9252.851224] Lustre: Skipped 1 previous similar message [ 9253.343179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9253.451686] Lustre: server umount lustre-OST0000 complete [ 9257.959481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.973188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9257.983751] Lustre: Skipped 1 previous similar message [ 9262.504991] Lustre: server umount lustre-MDT0000 complete [ 9263.073621] LustreError: 305644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.086895] LustreError: 305644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9265.718136] LustreError: 307599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790945 with bad export cookie 7924735861641991449 [ 9265.731192] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9265.734705] LustreError: 307599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9266.002994] Lustre: server umount lustre-MDT0001 complete [ 9274.641900] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:35:52 (1787790952) [ 9276.468919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9278.389380] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:35:55 (1787790955) [ 9280.899488] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9304.020655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.175238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9304.187308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9304.304211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.804572] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9315.206426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.280536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9315.321831] Lustre: 311459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9320.167462] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9341.038051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.220160] Lustre: Denying initial registration attempt from nid 192.168.201.146@tcp, specified as failover [ 9341.226435] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9341.236306] LustreError: 312912:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9341.248094] LustreError: 312912:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9341.251686] LustreError: 312912:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9341.323925] Lustre: server umount lustre-OST0000 complete [ 9341.328550] LustreError: 312912:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9346.528420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9346.533657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.541124] Lustre: Skipped 1 previous similar message [ 9346.546103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9352.738386] Lustre: server umount lustre-MDT0000 complete [ 9355.463833] LustreError: 311451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791035 with bad export cookie 7924735861641992331 [ 9355.466635] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9355.475214] LustreError: 311451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9355.823201] Lustre: server umount lustre-MDT0001 complete [ 9364.250417] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9366.447429] Key type lgssc unregistered [ 9366.702735] LNet: 314035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9366.710381] LNetError: 314035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9366.722597] LNet: Removed LNI 192.168.201.146@tcp [ 9367.402530] Key type .llcrypt unregistered [ 9367.404848] Key type ._llcrypt unregistered [ 9377.014628] Key type ._llcrypt registered [ 9377.016763] Key type .llcrypt registered [ 9377.071091] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9378.179757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9378.218760] alg: No test for adler32 (adler32-zlib) [ 9379.221438] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 9379.422895] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9381.055290] Key type lgssc registered [ 9381.938970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9422.307278] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9429.928836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9429.948489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.275344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9431.319385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9431.454701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9435.526808] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9442.897732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.987365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9443.005882] Lustre: 316779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9443.188588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9447.359756] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9454.191149] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9460.416250] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9479.275789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9479.452567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9479.612580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9485.285071] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9492.372434] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9495.523344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9495.535885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9495.548111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.032606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.048944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.064488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.076083] Lustre: Skipped 1 previous similar message [ 9500.495568] Lustre: server umount lustre-OST0000 complete [ 9504.737415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9504.749365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.764331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.124814] Lustre: server umount lustre-MDT0000 complete [ 9513.598519] LustreError: 316772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791193 with bad export cookie 14891435024373008280 [ 9513.599930] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.608879] LustreError: 316772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9513.895700] Lustre: server umount lustre-MDT0001 complete [ 9523.346683] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9525.942519] Key type lgssc unregistered [ 9526.231830] LNet: 320352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9526.237912] LNetError: 320352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9526.249657] LNet: Removed LNI 192.168.201.146@tcp [ 9527.139368] Key type .llcrypt unregistered [ 9527.146165] Key type ._llcrypt unregistered [ 9553.217280] Key type ._llcrypt registered [ 9553.220184] Key type .llcrypt registered [ 9553.361846] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9554.515770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9554.599914] alg: No test for adler32 (adler32-zlib) [ 9555.716303] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 9556.017684] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9557.775169] Key type lgssc registered [ 9558.737750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9565.667728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9565.679874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9567.079558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.366689] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9576.420234] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9576.430346] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9577.728070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9578.006368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9582.100394] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9589.142971] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9593.959360] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9601.773187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9602.067696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.204794] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9614.486433] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9624.032168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9624.036867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.046033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9626.592944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9626.594321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9626.611285] Lustre: Skipped 2 previous similar messages [ 9628.798554] Lustre: server umount lustre-MDT0000 complete [ 9631.717200] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9631.727631] LustreError: 321748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791311 with bad export cookie 3077951084496825552 [ 9631.734592] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9631.743818] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9631.747272] LustreError: 321748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9632.661823] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [ 9632.665043] Lustre: Skipped 1 previous similar message [ 9636.832972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.837120] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.844176] Lustre: Skipped 1 previous similar message [ 9636.852272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9636.869394] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9641.953505] LustreError: 321764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9641.953572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9641.977778] LustreError: 321764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9641.985511] Lustre: Skipped 1 previous similar message [ 9646.047149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9646.215211] Lustre: server umount lustre-MDT0001 complete [ 9651.627798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9659.968625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.776489] LustreError: 321079:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e293b58a x1874634982073088/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9677.010935] LustreError: 323600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.022214] LustreError: 323600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9677.088103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.132514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9680.595265] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9687.227889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.578779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9691.551611] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9691.715683] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9696.802278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9698.752312] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9700.309236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9706.097822] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9708.159750] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9713.122568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9713.124470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9713.129401] LustreError: Skipped 1 previous similar message [ 9713.132297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9713.148972] Lustre: Skipped 1 previous similar message [ 9713.151888] Lustre: Skipped 1 previous similar message [ 9718.991660] Lustre: server umount lustre-OST0000 complete [ 9722.340199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.361712] Lustre: Skipped 1 previous similar message [ 9732.577236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.584457] Lustre: Skipped 7 previous similar messages [ 9736.159901] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9736.335158] Lustre: server umount lustre-MDT0000 complete [ 9737.699786] LustreError: 325370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.707035] LustreError: 325370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9740.090602] LustreError: 321750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791419 with bad export cookie 3077951084496833224 [ 9740.094930] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.096886] LustreError: 321750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9740.336497] Lustre: server umount lustre-MDT0001 complete [ 9750.062699] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9752.532667] Key type lgssc unregistered [ 9752.839975] LNet: 328042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9752.852843] LNetError: 328042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9752.882731] LNet: Removed LNI 192.168.201.146@tcp [ 9753.816699] Key type .llcrypt unregistered [ 9753.819300] Key type ._llcrypt unregistered [ 9778.899533] Key type ._llcrypt registered [ 9778.902508] Key type .llcrypt registered [ 9779.020879] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [ 9779.901988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9779.944133] alg: No test for adler32 (adler32-zlib) [ 9781.055356] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [ 9781.418295] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [ 9783.127346] Key type lgssc registered [ 9784.295218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9791.417441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9791.443643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.020204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.027296] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9802.210196] LustreError: 329459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.227366] LustreError: 329459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9807.335096] LustreError: 329460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.669917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9810.375809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.594557] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9822.961075] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9828.856337] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9835.234684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.259107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.353843] Lustre: server umount lustre-MDT0000 complete [ 9840.609502] LustreError: 329460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.621601] LustreError: 329460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9841.770561] LustreError: 329441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791521 with bad export cookie 17072643547936020919 [ 9841.777160] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9841.779934] LustreError: 329441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9842.120741] Lustre: server umount lustre-MDT0001 complete [ 9855.105877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.407044] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9855.472405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9855.646626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9859.865899] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9867.757803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.825622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9867.854092] Lustre: 332049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9868.063578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9872.465564] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9879.702152] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9884.563973] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9893.222588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9893.375487] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9893.384105] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9893.393172] LustreError: 333930:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9893.403258] LustreError: 333930:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9893.406910] LustreError: 333930:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9893.466327] Lustre: server umount lustre-OST0000 complete [ 9893.469204] LustreError: 333930:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9902.203253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9902.463455] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9902.691911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.701480] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9908.318919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9913.676984] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9919.719126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9919.794696] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9919.846471] Lustre: lustre-OST0001: new disk, initializing [ 9919.848920] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9919.853176] Lustre: 335471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9919.932571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9924.823533] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [ 9927.811762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9927.826685] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9927.951070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9931.301345] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9938.400861] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9938.405270] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.416820] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9938.436748] Lustre: Skipped 2 previous similar messages [ 9938.457751] Lustre: Skipped 2 previous similar messages [ 9940.278351] Lustre: server umount lustre-OST0001 complete [ 9943.536784] LustreError: 334396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9943.550772] LustreError: 334396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9944.545899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9944.552856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.578359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9948.646207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9948.647557] LustreError: 334395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.657148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9948.698760] LustreError: 334395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9953.765324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9953.777370] Lustre: Skipped 2 previous similar messages [ 9958.370300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9958.610437] Lustre: server umount lustre-OST0000 complete [ 9964.001738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9964.006074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.019306] Lustre: Skipped 1 previous similar message [ 9968.808298] Lustre: server umount lustre-MDT0000 complete [ 9972.197230] LustreError: 332054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.215113] LustreError: 332054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9972.257150] LustreError: 332040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791652 with bad export cookie 17072643547936021472 [ 9972.263179] LustreError: 332040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9972.269036] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9972.709763] Lustre: server umount lustre-MDT0001 complete [ 9981.851710] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [ 9984.230750] Key type lgssc unregistered [ 9984.565526] LNet: 337433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9984.574805] LNetError: 337433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9984.599957] LNet: Removed LNI 192.168.201.146@tcp [ 9985.355887] Key type .llcrypt unregistered [ 9985.358330] Key type ._llcrypt unregistered [10046.250775] Key type ._llcrypt registered [10046.252221] Key type .llcrypt registered [10046.324434] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [10058.576995] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10059.597160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10059.611307] alg: No test for adler32 (adler32-zlib) [10060.693841] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [10060.903899] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10062.527160] Key type lgssc registered [10063.500475] Lustre: Echo OBD driver; http://www.lustre.org/ [10093.645175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10093.659649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.836398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10094.857972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10094.920286] Lustre: lustre-MDT0000: new disk, initializing [10094.977925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.994088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10098.493868] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10107.627919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.727830] Lustre: 342346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10107.769746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10107.772901] Lustre: Skipped 1 previous similar message [10107.823931] Lustre: lustre-MDT0001: new disk, initializing [10107.903244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10107.940764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10107.952261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10113.327390] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10121.931366] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10126.931102] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10132.425628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10132.684201] Lustre: lustre-OST0000: new disk, initializing [10132.688550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10132.695289] Lustre: 344310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10132.790613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.944432] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10142.256183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10142.266453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10142.329214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10146.323144] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10150.042647] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10150.238367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10154.227671] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10154.434893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10157.545456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10157.552549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.560662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10157.568168] Lustre: Skipped 1 previous similar message [10157.576827] Lustre: Skipped 1 previous similar message [10162.475596] Lustre: server umount lustre-OST0000 complete [10167.776598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.778471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10167.793079] Lustre: Skipped 1 previous similar message [10167.822115] Lustre: Skipped 1 previous similar message [10172.023159] Lustre: server umount lustre-MDT0000 complete [10176.094079] LustreError: 342338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791855 with bad export cookie 149368638435175233 [10176.097150] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.102306] LustreError: 342338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10176.451730] Lustre: server umount lustre-MDT0001 complete [10184.100077] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:51:01 (1787791861) [10190.733823] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:51:08 (1787791868) [10198.067313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.360373] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.368893] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10198.412974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10201.871594] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10203.618391] LustreError: 347067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.581900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.740982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10208.827418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10212.991418] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10219.740834] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10224.647742] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10232.441481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10233.047595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10239.762236] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10248.374497] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10256.176511] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10258.014501] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10259.770681] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10261.992418] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10264.045570] Lustre: Failing over lustre-MDT0000 [10265.059309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10265.073661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.097728] LustreError: 347067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.413715] Lustre: server umount lustre-MDT0000 complete [10269.154102] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10269.154884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.175915] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10269.181409] Lustre: Skipped 1 previous similar message [10274.275173] LustreError: 348306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.297850] LustreError: 348306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10282.383447] LustreError: 347063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.414264] LustreError: 347063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10285.282176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.384936] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.638096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10285.688364] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10286.464396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10289.836864] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10290.677445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10290.721674] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10290.780056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10298.464118] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10300.007850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10303.402680] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10305.005233] Lustre: Failing over lustre-MDT0000 [10305.453644] Lustre: server umount lustre-MDT0000 complete [10306.017923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.030162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10306.037601] Lustre: Skipped 1 previous similar message [10306.039370] LustreError: 347062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10306.039381] LustreError: 347062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10321.386766] Lustre: 340569:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787791985/real 1787791985] req@00000000498b78d2 x1874635511647872/t0(0) o400->MGC192.168.201.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787792001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10321.411687] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10324.221309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.631532] LustreError: 340567:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002395c13a x1874635511653376/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10331.927902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.972524] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10333.089732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10335.478957] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10337.265066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10337.270867] Lustre: Skipped 2 previous similar messages [10337.306234] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10337.358915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10343.899337] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10345.408816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10352.611482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.614671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.618838] Lustre: Skipped 2 previous similar messages [10352.634589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10356.237719] Lustre: server umount lustre-MDT0000 complete [10357.732971] LustreError: 347063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.762595] LustreError: 347063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [10359.501823] LustreError: 348907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792039 with bad export cookie 149368638435177949 [10359.508883] LustreError: 348907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10359.510751] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10359.740025] Lustre: server umount lustre-MDT0001 complete [10373.953910] Lustre: server umount lustre-OST0000 complete [10390.039635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.825477] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10402.217639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.633062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10402.640411] Lustre: Skipped 1 previous similar message [10406.533902] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10415.105331] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10424.933630] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10433.363600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.709579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10438.895671] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10445.702774] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10456.544025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10456.552555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.559283] Lustre: Skipped 1 previous similar message [10456.563993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10456.569020] Lustre: Skipped 2 previous similar messages [10468.835318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10468.841739] Lustre: Skipped 4 previous similar messages [10469.343267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10469.507936] Lustre: server umount lustre-OST0000 complete [10487.263792] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10487.370366] Lustre: server umount lustre-MDT0000 complete [10489.312307] LustreError: 353406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.322829] LustreError: 353406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10492.206046] LustreError: 353387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792172 with bad export cookie 149368638435178684 [10492.212218] LustreError: 353387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10492.212415] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.479962] Lustre: server umount lustre-MDT0001 complete [10500.661901] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [10503.133082] Key type lgssc unregistered [10503.421602] LNet: 356941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10503.425072] LNetError: 356941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10503.454089] LNet: Removed LNI 192.168.201.146@tcp [10504.244862] Key type .llcrypt unregistered [10504.247197] Key type ._llcrypt unregistered [10524.916245] Key type ._llcrypt registered [10524.917980] Key type .llcrypt registered [10524.994969] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10525.857829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10525.994669] alg: No test for adler32 (adler32-zlib) [10526.946900] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [10527.112050] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10528.753417] Key type lgssc registered [10529.518928] Lustre: Echo OBD driver; http://www.lustre.org/ [10536.596536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10536.613263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.983509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10542.598282] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10547.170931] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.180427] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10551.773745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.034734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10556.509786] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10564.080979] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10568.963083] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10576.517949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10576.759987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10582.934983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10584.505676] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10593.389267] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10608.611868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10608.633847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10608.647603] Lustre: Skipped 2 previous similar messages [10608.666384] Lustre: Skipped 2 previous similar messages [10613.051401] Lustre: server umount lustre-MDT0000 complete [10613.731752] LustreError: 358435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.748572] LustreError: 358435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10616.713444] LustreError: 358418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792296 with bad export cookie 9841052210181033923 [10616.731668] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.986501] Lustre: server umount lustre-MDT0001 complete [10631.449326] Lustre: server umount lustre-OST0000 complete [10643.849339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.246894] LustreError: 362169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10644.301585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10648.599757] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10649.570071] LustreError: 362170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10654.702829] LustreError: 362169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10656.555466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.842923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10660.325259] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10666.746728] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10672.866055] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10679.481895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10679.790565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10685.775600] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10690.043918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10693.188848] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10707.938027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10707.944561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.959809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10710.499801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10710.500936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10710.517524] Lustre: Skipped 1 previous similar message [10710.573106] Lustre: Skipped 1 previous similar message [10712.039717] Lustre: server umount lustre-MDT0000 complete [10715.622370] LustreError: 362164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.639162] LustreError: 362164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10716.786759] LustreError: 362150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792396 with bad export cookie 9841052210181035036 [10716.794723] LustreError: 362150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10716.795553] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10717.351953] Lustre: server umount lustre-MDT0001 complete [10732.028871] Lustre: server umount lustre-OST0000 complete [10748.002239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.396518] LustreError: 366084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.450991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10753.414710] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10761.454211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.149691] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10772.227590] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10777.574704] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10784.746210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10785.026222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10785.029898] Lustre: Skipped 1 previous similar message [10792.051969] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [10793.202453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10798.636535] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10803.682201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10803.689395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.720146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.725482] Lustre: Skipped 1 previous similar message [10805.807794] Lustre: server umount lustre-OST0000 complete [10813.409671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.418751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.427554] Lustre: Skipped 1 previous similar message [10813.435803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.437562] Lustre: Skipped 1 previous similar message [10815.556563] Lustre: server umount lustre-MDT0000 complete [10819.022736] LustreError: 366065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792498 with bad export cookie 9841052210181036065 [10819.035891] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10819.043031] LustreError: 366065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10819.660842] Lustre: server umount lustre-MDT0001 complete [10830.433231] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [10833.039356] Key type lgssc unregistered [10833.288850] LNet: 369629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10833.302607] LNetError: 369629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10833.318750] LNet: Removed LNI 192.168.201.146@tcp [10834.217851] Key type .llcrypt unregistered [10834.221845] Key type ._llcrypt unregistered [10863.456962] Key type ._llcrypt registered [10863.459701] Key type .llcrypt registered [10863.567409] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10864.819669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10864.864896] alg: No test for adler32 (adler32-zlib) [10865.899244] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [10866.084958] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10867.816585] Key type lgssc registered [10868.930149] Lustre: Echo OBD driver; http://www.lustre.org/ [10876.663952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10876.676951] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.681876] LustreError: 371073:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10876.696267] LustreError: 371073:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10876.701889] LustreError: 371073:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10876.707236] LustreError: 371073:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10876.714217] LustreError: 371073:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10885.168204] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10885.182350] LustreError: 371470:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10885.229735] LustreError: 371470:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10885.242171] LustreError: 371470:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10885.256398] LustreError: 371470:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10885.265953] LustreError: 371470:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10905.707390] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [10908.557530] Key type lgssc unregistered [10908.879582] LNet: 372479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10908.883938] LNetError: 372479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10908.917938] LNet: Removed LNI 192.168.201.146@tcp [10909.889564] Key type .llcrypt unregistered [10909.893801] Key type ._llcrypt unregistered [10928.694284] Key type ._llcrypt registered [10928.697745] Key type .llcrypt registered [10928.811464] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [10945.732453] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [10946.936525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10946.968451] alg: No test for adler32 (adler32-zlib) [10948.006937] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [10948.245576] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [10949.935212] Key type lgssc registered [10951.319044] Lustre: Echo OBD driver; http://www.lustre.org/ [10990.785861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10990.837209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.215606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10992.252659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10992.396382] Lustre: lustre-MDT0000: new disk, initializing [10992.486815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10992.509366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10998.265487] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11009.519623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.699517] Lustre: 375640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11009.822606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11009.828824] Lustre: Skipped 1 previous similar message [11010.002891] Lustre: lustre-MDT0001: new disk, initializing [11010.233996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11010.327646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11010.348830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11016.736589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11028.228412] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11034.599837] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11042.594923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11042.921055] Lustre: lustre-OST0000: new disk, initializing [11042.930143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11042.937846] Lustre: 377609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11043.087393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11049.176789] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11051.157994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11051.181376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11051.302887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11057.839509] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11062.824356] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11063.061537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11067.781660] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11068.088393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11071.969060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11071.976495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11071.988693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11077.092124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11077.099735] Lustre: Skipped 2 previous similar messages [11082.210765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11082.223759] Lustre: Skipped 1 previous similar message [11085.279128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11085.507508] Lustre: server umount lustre-OST0000 complete [11091.936334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11091.946586] LustreError: Skipped 1 previous similar message [11091.950121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.970985] Lustre: Skipped 1 previous similar message [11091.985033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11095.914106] Lustre: server umount lustre-MDT0000 complete [11100.063844] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792779 with bad export cookie 12081364169423609191 [11100.071296] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.080100] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11100.580256] Lustre: server umount lustre-MDT0001 complete [11110.574313] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11112.206091] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:06:29 (1787792789) [11118.916174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.480304] LustreError: 380068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.503093] LustreError: 380068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11119.567909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11124.707920] LustreError: 380069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.848899] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11129.826909] LustreError: 380068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.438374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.737768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11138.494421] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11147.500145] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11155.479362] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11162.442686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.059525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11169.775868] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11178.551708] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11186.134950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.461591] Lustre: lustre-OST0001: new disk, initializing [11186.465591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11186.471352] Lustre: 382907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11186.571152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11193.442356] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11195.115978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11195.129179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11195.250492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11204.140892] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11210.751546] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11210.755538] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.790701] Lustre: Skipped 1 previous similar message [11210.811300] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11210.814196] Lustre: Skipped 2 previous similar messages [11214.734796] Lustre: server umount lustre-OST0001 complete [11215.848907] LustreError: 383337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.865661] LustreError: 383337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11220.968710] LustreError: 383337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.992969] LustreError: 383337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11231.215774] LustreError: 381892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11231.229740] LustreError: 381892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11234.784494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11234.797075] LustreError: Skipped 1 previous similar message [11234.799039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.824376] Lustre: Skipped 1 previous similar message [11234.843829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11234.855302] Lustre: Skipped 1 previous similar message [11238.911710] Lustre: server umount lustre-OST0000 complete [11246.560886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11246.567214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11246.592841] Lustre: Skipped 2 previous similar messages [11249.904071] Lustre: server umount lustre-MDT0000 complete [11251.680608] LustreError: 380068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.696743] LustreError: 380068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11253.870070] LustreError: 381913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792933 with bad export cookie 12081364169423610073 [11253.886242] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11253.891383] LustreError: 381913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11260.467679] Lustre: server umount lustre-MDT0001 complete [11272.041421] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11275.759798] Key type lgssc unregistered [11276.256943] LNet: 384899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11276.265923] LNetError: 384899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11276.278866] LNet: Removed LNI 192.168.201.146@tcp [11277.575873] Key type .llcrypt unregistered [11277.578177] Key type ._llcrypt unregistered [11342.107987] Key type ._llcrypt registered [11342.109915] Key type .llcrypt registered [11342.298714] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11344.428020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11344.452687] alg: No test for adler32 (adler32-zlib) [11345.609207] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [11346.110862] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11347.999167] Key type lgssc registered [11349.624919] Lustre: Echo OBD driver; http://www.lustre.org/ [11357.974546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11358.012709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11359.498991] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11359.555811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11359.773531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11363.546338] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11372.905471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.030666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11373.079644] Lustre: 387698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11373.521593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11379.296502] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11387.720761] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11393.426713] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11399.138889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11399.152852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.173135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11400.161738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11400.165956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11402.246292] Lustre: server umount lustre-MDT0000 complete [11405.691868] LustreError: 387691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793085 with bad export cookie 2628860789592960163 [11405.694706] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11405.714043] LustreError: 387691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11406.023884] Lustre: server umount lustre-MDT0001 complete [11414.967557] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:11:32 (1787793092) [11421.601698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11422.300036] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.325959] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11422.399613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11427.506052] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11427.815971] LustreError: 390211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.928627] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.476299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.674984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11441.698726] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11449.488925] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11455.892400] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11463.612458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.876590] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11464.224633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11471.618791] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11480.451037] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11488.392144] Lustre: Permanently deactivating lustre-OST0000 [11488.428938] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11497.538906] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11498.955686] LustreError: 393199:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11498.969378] LustreError: 393199:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11498.980434] LustreError: 393199:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11503.526458] LustreError: 393248:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11503.545280] LustreError: 393248:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11512.225855] Lustre: server umount lustre-OST0000 complete [11515.066687] LustreError: 393497:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11515.073253] LustreError: 393497:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11518.436272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11518.453757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11518.475180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.480365] Lustre: Skipped 1 previous similar message [11524.578731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11524.588851] Lustre: Skipped 3 previous similar messages [11529.696734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11531.231157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11531.430384] Lustre: server umount lustre-MDT0000 complete [11534.818281] LustreError: 390936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.841202] LustreError: 390936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11535.371718] LustreError: 390192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793215 with bad export cookie 2628860789592960744 [11535.384443] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11535.386257] LustreError: 390192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11545.719001] Lustre: server umount lustre-MDT0001 complete [11556.026645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11560.720729] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11563.433554] LustreError: 394658:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11569.409624] Lustre: 394998:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.146@tcp [11571.199059] Lustre: server umount MGS complete [11582.764113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.261187] LustreError: 395723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.278778] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11583.283500] Lustre: Skipped 1 previous similar message [11583.343344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.233576] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11593.696989] LustreError: 395724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.717211] LustreError: 395724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11597.328162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11597.660540] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11602.705918] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11610.761323] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11618.429762] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11620.697750] Lustre: Permanently reactivating lustre-OST0000 [11620.727379] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11620.730360] Lustre: Skipped 2 previous similar messages [11649.504151] LustreError: 397512:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11657.141247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.731570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11657.742278] Lustre: Skipped 1 previous similar message [11664.816593] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11673.426932] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11679.209802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11679.217618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.243466] Lustre: Skipped 1 previous similar message [11679.248413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11679.250882] Lustre: Skipped 1 previous similar message [11685.390877] Lustre: server umount lustre-OST0000 complete [11689.952376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11689.959927] LustreError: Skipped 1 previous similar message [11689.961364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.968859] Lustre: Skipped 1 previous similar message [11695.584614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11695.590359] Lustre: Skipped 6 previous similar messages [11703.266027] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11703.458719] Lustre: server umount lustre-MDT0000 complete [11705.832506] LustreError: 396451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.849303] LustreError: 396451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11707.687988] LustreError: 395704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793387 with bad export cookie 2628860789592961794 [11707.701444] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11707.705406] LustreError: 395704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11708.059444] Lustre: server umount lustre-MDT0001 complete [11723.177773] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [11726.540865] Key type lgssc unregistered [11726.840836] LNet: 400091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11726.846595] LNetError: 400091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11726.865045] LNet: Removed LNI 192.168.201.146@tcp [11727.934855] Key type .llcrypt unregistered [11727.937326] Key type ._llcrypt unregistered [11750.004718] Key type ._llcrypt registered [11750.009040] Key type .llcrypt registered [11750.172827] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [11769.998395] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [11772.366966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11772.481607] alg: No test for adler32 (adler32-zlib) [11773.693699] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [11774.082388] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [11775.863489] Key type lgssc registered [11777.688033] Lustre: Echo OBD driver; http://www.lustre.org/ [11815.846949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11815.893490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11817.303769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11817.334679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11817.430596] Lustre: lustre-MDT0000: new disk, initializing [11817.534741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11817.554190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11823.213736] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11835.652932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.867111] Lustre: 403248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11835.971938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11835.979337] Lustre: Skipped 1 previous similar message [11836.050334] Lustre: lustre-MDT0001: new disk, initializing [11836.145462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11836.188865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11836.199773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11841.990761] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11853.355600] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11860.188466] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11868.340596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11868.658363] Lustre: lustre-OST0000: new disk, initializing [11868.661440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11868.666368] Lustre: 405219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11868.759842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11874.755624] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11875.410129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11875.424531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11875.533703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11885.078942] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11889.727234] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11889.994568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11894.746595] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11895.042991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11901.410392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11901.415078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11901.445740] Lustre: Skipped 1 previous similar message [11904.033687] Lustre: server umount lustre-OST0000 complete [11911.658752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.669656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.671852] Lustre: Skipped 1 previous similar message [11911.689070] Lustre: Skipped 2 previous similar messages [11916.517744] Lustre: server umount lustre-MDT0000 complete [11921.035774] LustreError: 403242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793600 with bad export cookie 1351303089464141947 [11921.044959] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11921.531553] Lustre: server umount lustre-MDT0001 complete [11932.314221] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11934.406430] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:20:11 (1787793611) [11941.195124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.722949] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11941.740615] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11941.782464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11946.323921] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11946.982730] LustreError: 407685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.097403] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.106191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.618450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11960.208962] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11968.100941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11973.771753] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11980.658581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11980.905336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11987.259552] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [11997.467592] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11999.522220] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12003.245896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12010.977290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12010.986901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.013902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12011.058230] Lustre: Skipped 1 previous similar message [12014.515578] Lustre: server umount lustre-OST0000 complete [12021.216837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12021.228971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12021.254664] Lustre: Skipped 1 previous similar message [12021.261148] Lustre: Skipped 1 previous similar message [12024.931788] Lustre: server umount lustre-MDT0000 complete [12026.336220] LustreError: 408924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.369713] LustreError: 408924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12028.786871] LustreError: 410369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793708 with bad export cookie 1351303089464142829 [12028.787859] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.802255] LustreError: 410369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12029.161559] Lustre: server umount lustre-MDT0001 complete [12041.345666] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12044.718888] Key type lgssc unregistered [12045.147373] LNet: 411413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12045.158024] LNetError: 411413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12045.191057] LNet: Removed LNI 192.168.201.146@tcp [12046.782233] Key type .llcrypt unregistered [12046.791432] Key type ._llcrypt unregistered [12078.189796] Key type ._llcrypt registered [12078.192651] Key type .llcrypt registered [12078.297315] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12078.717331] Key type .llcrypt unregistered [12078.719914] Key type ._llcrypt unregistered [12097.919789] Key type ._llcrypt registered [12097.922269] Key type .llcrypt registered [12098.151383] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12099.200654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12099.232290] alg: No test for adler32 (adler32-zlib) [12100.414986] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12100.766276] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12102.567475] Key type lgssc registered [12103.723724] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.270084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12113.314081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12114.702035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12119.474130] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12125.161648] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.181755] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12128.113344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12128.443560] LustreError: 413632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.466160] LustreError: 413632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12128.485721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12133.858189] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12133.871719] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12134.604278] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12138.983831] LustreError: 413632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.004995] LustreError: 413632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12141.050446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12141.982316] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12146.152381] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.169400] LustreError: 413631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12149.053524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.485050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12154.476658] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12165.108897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.130519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12165.132749] LustreError: 414355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.319289] Lustre: server umount lustre-OST0000 complete [12169.544714] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12169.556438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.572739] Lustre: Skipped 1 previous similar message [12169.577729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12170.214824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12170.224707] Lustre: Skipped 1 previous similar message [12175.329227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.343142] Lustre: Skipped 1 previous similar message [12175.803515] Lustre: server umount lustre-MDT0000 complete [12179.371915] LustreError: 413612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793859 with bad export cookie 11440776232501628800 [12179.377609] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12179.382817] LustreError: 413612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12179.750886] Lustre: server umount lustre-MDT0001 complete [12193.019686] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12196.992845] Key type lgssc unregistered [12197.320897] LNet: 416782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12197.334332] LNetError: 416782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12197.347971] LNet: Removed LNI 192.168.201.146@tcp [12198.518697] Key type .llcrypt unregistered [12198.521784] Key type ._llcrypt unregistered [12226.848425] Key type ._llcrypt registered [12226.850973] Key type .llcrypt registered [12227.024370] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12228.110587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12228.224985] alg: No test for adler32 (adler32-zlib) [12229.415117] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12229.714074] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12231.391306] Key type lgssc registered [12233.079805] Lustre: Echo OBD driver; http://www.lustre.org/ [12240.820830] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12240.852799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.415660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12261.537300] LustreError: 418175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.146@tcp: failed processing log, type 4: rc = -110 [12288.223249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12294.431981] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12301.976078] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12308.944622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.473917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12309.543276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12314.229388] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12314.598556] LustreError: 418200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12314.621839] LustreError: 418200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12319.717689] LustreError: 418199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.787318] LustreError: 418200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.799535] LustreError: 418200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12323.497389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12323.953875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12328.285288] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12335.548295] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12341.200537] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12349.411809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12349.414990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.447661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12349.926740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.963797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12350.395714] Lustre: server umount lustre-OST0000 complete [12354.536580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12354.544162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.560298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12356.877811] Lustre: server umount lustre-MDT0000 complete [12360.164036] LustreError: 419211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.189535] LustreError: 419211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12360.471354] LustreError: 418183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794040 with bad export cookie 3278023022549895224 [12360.476112] LustreError: 418183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12360.477541] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12360.638830] Lustre: server umount lustre-MDT0001 complete [12373.390563] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12376.341805] Key type lgssc unregistered [12376.484716] LNet: 421743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12376.489075] LNetError: 421743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12376.504893] LNet: Removed LNI 192.168.201.146@tcp [12377.184468] Key type .llcrypt unregistered [12377.186122] Key type ._llcrypt unregistered [12401.224282] Key type ._llcrypt registered [12401.228446] Key type .llcrypt registered [12401.416596] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12402.405290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12402.471790] alg: No test for adler32 (adler32-zlib) [12403.735161] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12404.007830] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12405.687250] Key type lgssc registered [12407.037499] Lustre: Echo OBD driver; http://www.lustre.org/ [12414.244936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12414.262495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12415.743638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12420.257886] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12425.192238] LustreError: 423159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.214502] LustreError: 423159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12429.281685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.724451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12434.805848] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12442.510675] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12449.571331] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12458.726605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.337325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12461.366750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12467.229514] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12477.276751] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12485.093428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.097598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12485.101517] Lustre: Skipped 2 previous similar messages [12485.109933] Lustre: Skipped 2 previous similar messages [12488.857683] Lustre: server umount lustre-MDT0000 complete [12490.214813] LustreError: 423155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.219920] LustreError: 423155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12491.417904] LustreError: 423140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.46@tcp arrived at 1787794171 with bad export cookie 3084343309738403056 [12491.435581] LustreError: 423140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12493.521654] LustreError: 423139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794173 with bad export cookie 3084343309738402160 [12493.532152] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12493.544465] LustreError: 423139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12495.328716] LustreError: 423158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.336727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12495.337266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12495.347811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12495.347818] Lustre: Skipped 1 previous similar message [12495.349618] LustreError: 423158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12498.407673] LustreError: 423159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12498.409666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12498.439232] LustreError: 423159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12499.923325] Lustre: server umount lustre-OST0000 complete [12503.520387] LustreError: 423158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.543800] LustreError: 423158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12506.929743] Lustre: server umount lustre-MDT0001 complete [12519.874434] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12523.154114] Key type lgssc unregistered [12523.452912] LNet: 426847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12523.457978] LNetError: 426847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12523.478184] LNet: Removed LNI 192.168.201.146@tcp [12524.626547] Key type .llcrypt unregistered [12524.629936] Key type ._llcrypt unregistered [12551.362778] Key type ._llcrypt registered [12551.364604] Key type .llcrypt registered [12551.488596] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12552.739143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12552.888646] alg: No test for adler32 (adler32-zlib) [12553.916303] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12554.226685] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12555.935183] Key type lgssc registered [12557.239270] Lustre: Echo OBD driver; http://www.lustre.org/ [12565.399362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12565.419082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.836080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.930633] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12576.226589] LustreError: 428268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.242263] LustreError: 428268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12579.002169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.373683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12583.646898] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12591.463157] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12596.641736] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12605.503607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12606.073752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12611.202984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12615.409241] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12623.648449] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12629.992716] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.005049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12630.015953] Lustre: Skipped 1 previous similar message [12631.478807] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12634.662273] Lustre: server umount lustre-MDT0001 complete [12635.112947] LustreError: 428268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12635.145069] LustreError: 428268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12636.547201] LustreError: 428263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12640.231979] LustreError: 428267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12640.246094] LustreError: 428267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12642.272533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12642.286317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12642.302965] Lustre: Skipped 1 previous similar message [12642.312080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12645.362797] LustreError: 429281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.366353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12647.940602] Lustre: server umount lustre-OST0000 complete [12652.391437] Lustre: server umount lustre-MDT0000 complete [12668.162274] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12671.447722] Key type lgssc unregistered [12671.938180] LNet: 431955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12671.944422] LNetError: 431955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12671.961748] LNet: Removed LNI 192.168.201.146@tcp [12673.052779] Key type .llcrypt unregistered [12673.054478] Key type ._llcrypt unregistered [12701.696305] Key type ._llcrypt registered [12701.702186] Key type .llcrypt registered [12701.821366] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [12717.210851] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12718.524820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12718.702579] alg: No test for adler32 (adler32-zlib) [12719.878169] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12720.162212] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12721.927197] Key type lgssc registered [12723.189473] Lustre: Echo OBD driver; http://www.lustre.org/ [12763.674566] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12779.706209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12779.746639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12781.105341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12781.137567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12781.229787] Lustre: lustre-MDT0000: new disk, initializing [12781.307459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12781.337357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12785.754687] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12802.294967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12802.636928] Lustre: lustre-OST0000: new disk, initializing [12802.642909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12802.646841] Lustre: Skipped 1 previous similar message [12802.655311] Lustre: 436992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12802.750485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12808.654655] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12808.788105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12808.800739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12808.836916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12818.699941] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12833.925478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12834.071214] Lustre: 435972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12834.208183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12834.343435] Lustre: lustre-MDT0001: new disk, initializing [12834.510575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12834.593156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12834.612148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12839.134162] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [12848.706320] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12849.017356] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12853.878387] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12854.056867] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12861.866306] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12861.931130] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.937230] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12866.944133] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12866.947442] Lustre: Skipped 2 previous similar messages [12872.070322] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.46@tcp (stopping) [12872.091051] Lustre: Skipped 2 previous similar messages [12873.183177] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12873.535276] Lustre: server umount lustre-MDT0001 complete [12882.409068] LustreError: 435980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12882.438561] LustreError: 435980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12887.524083] LustreError: 435982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12887.544481] LustreError: 435982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12892.642211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12892.642507] LustreError: 438667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12892.665930] Lustre: Skipped 1 previous similar message [12892.668090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12892.716882] Lustre: Skipped 2 previous similar messages [12897.442380] Lustre: server umount lustre-OST0000 complete [12897.760955] LustreError: 435986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.167126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12915.704779] Lustre: server umount lustre-MDT0000 complete [12928.387258] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12931.049827] Key type lgssc unregistered [12931.296846] LNet: 440745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12931.307511] LNetError: 440745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12931.331635] LNet: Removed LNI 192.168.201.146@tcp [12932.168857] Key type .llcrypt unregistered [12932.171426] Key type ._llcrypt unregistered [12967.450310] Key type ._llcrypt registered [12967.453946] Key type .llcrypt registered [12967.554546] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [12967.877827] Key type .llcrypt unregistered [12967.882720] Key type ._llcrypt unregistered [12983.097361] Key type ._llcrypt registered [12983.098739] Key type .llcrypt registered [12983.173813] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [12984.317795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12984.339276] alg: No test for adler32 (adler32-zlib) [12985.354658] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [12985.508483] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [12987.168887] Key type lgssc registered [12988.237388] Lustre: Echo OBD driver; http://www.lustre.org/ [13014.347514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13014.403933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13015.544665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13015.562356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13015.610690] Lustre: lustre-MDT0000: new disk, initializing [13015.677702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13015.695298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13020.668056] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13031.337589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13031.507499] Lustre: 444124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13031.567703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13031.580519] Lustre: Skipped 1 previous similar message [13031.678515] Lustre: lustre-MDT0001: new disk, initializing [13031.849057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13031.913178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13031.926430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13037.188850] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13048.217645] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13053.906182] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13060.619901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13060.957934] Lustre: lustre-OST0000: new disk, initializing [13060.964666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13060.970846] Lustre: 446093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13061.078618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13067.545132] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13069.914542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13069.926292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13070.030355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13077.107493] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13085.674028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.681316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13085.690703] Lustre: Skipped 2 previous similar messages [13085.701030] Lustre: Skipped 2 previous similar messages [13090.243835] Lustre: server umount lustre-MDT0000 complete [13094.458029] LustreError: 444117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794774 with bad export cookie 15771922530991868429 [13094.474865] LustreError: 444117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13094.476214] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13095.095572] Lustre: server umount lustre-MDT0001 complete [13109.373963] Lustre: server umount lustre-OST0000 complete [13118.686655] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:39:56 (1787794796) [13125.364163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13125.648358] LustreError: 448123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13125.675144] LustreError: 448123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13125.708298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13129.266749] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13130.721875] LustreError: 448124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.841968] LustreError: 448123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13136.893750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13137.202185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13141.365379] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13149.118079] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13154.934456] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13174.589093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13174.848632] Lustre: Found index 0 for lustre-OST0000, updating log [13175.147650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13181.754741] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13189.732609] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13196.674934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13207.011595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13207.017285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13207.022034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13207.048691] Lustre: Skipped 1 previous similar message [13207.057190] Lustre: Skipped 1 previous similar message [13208.127128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13208.139428] Lustre: Skipped 1 previous similar message [13214.177040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13214.199485] Lustre: Skipped 1 previous similar message [13219.304622] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13219.571193] Lustre: server umount lustre-OST0000 complete [13235.423354] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [13235.681059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13235.692104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13235.701914] Lustre: Skipped 1 previous similar message [13241.719107] Lustre: server umount lustre-MDT0000 complete [13245.556751] LustreError: 448103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794925 with bad export cookie 15771922530991869822 [13245.562775] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13245.567429] LustreError: 448103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13246.083335] Lustre: server umount lustre-MDT0001 complete [13255.312680] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:42:12 (1787794932) [13878.339839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13878.774930] LustreError: 454559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.847803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13882.938670] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13883.874302] LustreError: 454560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13889.004676] LustreError: 454559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13893.607385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13893.958728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13898.360395] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13904.952543] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13910.228623] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13928.965258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13929.311210] Lustre: Found index 0 for lustre-OST0000, updating log [13929.680941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13935.610827] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [13940.964212] LNet: 1 local NIs in recovery (showing 1): 192.168.201.146@tcp [13942.839261] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13944.764518] Lustre: server umount lustre-OST0000 complete [13949.162547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13949.170339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13949.181019] Lustre: Skipped 2 previous similar messages [13949.187796] Lustre: Skipped 3 previous similar messages [13950.241358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13950.248360] Lustre: Skipped 1 previous similar message [13952.289334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13952.294445] Lustre: Skipped 3 previous similar messages [13954.368750] LustreError: 454559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13954.383444] LustreError: 454559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13954.443706] Lustre: server umount lustre-MDT0000 complete [13957.901717] LustreError: 454539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795637 with bad export cookie 15771922530991870907 [13957.911425] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13958.269558] Lustre: server umount lustre-MDT0001 complete [13988.250958] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [13999.238271] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14034.195501] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:55:11 (1787795711) [14052.474222] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14055.585187] Key type lgssc unregistered [14055.918541] LNet: 461174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14055.929378] LNetError: 461174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14181.951770] LNet: Removed LNI 192.168.201.146@tcp [14183.084689] Key type .llcrypt unregistered [14183.086751] Key type ._llcrypt unregistered [14201.789991] Key type ._llcrypt registered [14201.791993] Key type .llcrypt registered [14201.884599] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14203.025406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14203.069904] alg: No test for adler32 (adler32-zlib) [14204.167909] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [14204.460949] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14206.143235] Key type lgssc registered [14207.188679] Lustre: Echo OBD driver; http://www.lustre.org/ [14211.973715] LNet: Added LNI 192.168.201.146@tcp100 [8/256/0/180] [14212.718682] LNet: Added LNI 192.168.201.146@tcp101 [8/256/0/180] [14213.638210] LNet: Added LNI 192.168.201.146@tcp102 [8/256/0/180] [14214.377503] LNet: Added LNI 192.168.201.146@tcp103 [8/256/0/180] [14215.028744] LNet: Added LNI 192.168.201.146@tcp104 [8/256/0/180] [14215.912873] LNet: Added LNI 192.168.201.146@tcp105 [8/256/0/180] [14216.819615] LNet: Added LNI 192.168.201.146@tcp106 [8/256/0/180] [14217.599577] LNet: Added LNI 192.168.201.146@tcp107 [8/256/0/180] [14218.350918] LNet: Added LNI 192.168.201.146@tcp108 [8/256/0/180] [14219.150855] LNet: Added LNI 192.168.201.146@tcp109 [8/256/0/180] [14219.839885] LNet: Added LNI 192.168.201.146@tcp110 [8/256/0/180] [14220.572039] LNet: Added LNI 192.168.201.146@tcp111 [8/256/0/180] [14221.590075] LNet: Added LNI 192.168.201.146@tcp112 [8/256/0/180] [14222.683655] LNet: Added LNI 192.168.201.146@tcp113 [8/256/0/180] [14223.507359] LNet: Added LNI 192.168.201.146@tcp114 [8/256/0/180] [14224.573743] LNet: Added LNI 192.168.201.146@tcp115 [8/256/0/180] [14225.624342] LNet: Added LNI 192.168.201.146@tcp116 [8/256/0/180] [14226.668860] LNet: Added LNI 192.168.201.146@tcp117 [8/256/0/180] [14228.149390] LNet: Added LNI 192.168.201.146@tcp118 [8/256/0/180] [14229.355816] LNet: Added LNI 192.168.201.146@tcp119 [8/256/0/180] [14230.515068] LNet: Added LNI 192.168.201.146@tcp120 [8/256/0/180] [14231.916793] LNet: Added LNI 192.168.201.146@tcp121 [8/256/0/180] [14233.148820] LNet: Added LNI 192.168.201.146@tcp122 [8/256/0/180] [14234.372041] LNet: Added LNI 192.168.201.146@tcp123 [8/256/0/180] [14235.678775] LNet: Added LNI 192.168.201.146@tcp124 [8/256/0/180] [14237.066622] LNet: Added LNI 192.168.201.146@tcp125 [8/256/0/180] [14238.240063] LNet: Added LNI 192.168.201.146@tcp126 [8/256/0/180] [14239.386115] LNet: Added LNI 192.168.201.146@tcp127 [8/256/0/180] [14240.483261] LNet: Added LNI 192.168.201.146@tcp128 [8/256/0/180] [14241.514574] LNet: Added LNI 192.168.201.146@tcp129 [8/256/0/180] [14242.494988] LNet: Added LNI 192.168.201.146@tcp130 [8/256/0/180] [14243.484303] LNet: Added LNI 192.168.201.146@tcp131 [8/256/0/180] [14244.528144] LNet: Added LNI 192.168.201.146@tcp132 [8/256/0/180] [14245.571163] LNet: Added LNI 192.168.201.146@tcp133 [8/256/0/180] [14246.528161] LNet: Added LNI 192.168.201.146@tcp134 [8/256/0/180] [14247.396132] LNet: Added LNI 192.168.201.146@tcp135 [8/256/0/180] [14255.102074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14255.140063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14256.299310] Lustre: 464209:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14256.411466] LNetError: 464233:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14256.422589] LNetError: Protocol error connecting to 192.168.201.146@tcp114 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14256.563683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14256.618638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14256.703876] Lustre: lustre-MDT0000: new disk, initializing [14256.783104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14256.800716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14257.443336] LNetError: 461669:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14257.467991] LNetError: Protocol error connecting to 192.168.201.146@tcp113 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14262.198543] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14273.109717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14273.231542] Lustre: 464227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14273.281732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14273.284727] Lustre: Skipped 1 previous similar message [14273.397869] Lustre: lustre-MDT0001: new disk, initializing [14273.448714] Lustre: 465025:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14273.462374] Lustre: 465025:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14273.650953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14273.682373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14273.695469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14278.786335] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14288.332167] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14292.515346] LNetError: 464229:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14292.539499] LNetError: Protocol error connecting to 192.168.201.146@tcp113 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14294.075566] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14297.635128] LNetError: 464236:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14297.647896] LNetError: Protocol error connecting to 192.168.201.146@tcp114 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14300.684583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14300.994437] Lustre: lustre-OST0000: new disk, initializing [14300.997727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14301.003380] Lustre: 466199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14301.105170] Lustre: 466204:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14301.111730] Lustre: 466204:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14301.156785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14304.800561] LNetError: 464230:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14304.804686] LNetError: Protocol error connecting to 192.168.201.146@tcp113 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14306.802708] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14306.837069] Lustre: 466552:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.146@tcp: can't add NID 192.168.201.146@tcp130: rc = -75 [14306.848389] Lustre: 466552:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14306.930808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14306.942535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14307.087385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14316.966161] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14317.089103] LNetError: 461668:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14317.105309] LNetError: 461668:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14317.117837] LNetError: Protocol error connecting to 192.168.201.146@tcp113 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14317.142400] LNetError: Skipped 1 previous similar message [14327.776322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14327.779483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14327.783287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14327.790274] Lustre: Skipped 1 previous similar message [14327.803934] Lustre: Skipped 1 previous similar message [14330.839843] Lustre: server umount lustre-OST0000 complete [14334.944173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14334.948194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14334.956891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14335.587599] LNetError: 464239:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.146 [14335.602458] LNetError: 464239:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14335.608871] LNetError: Protocol error connecting to 192.168.201.146@tcp114 at host 192.168.201.146:988: is it running a compatible version of Lustre? [14335.619343] LNetError: Skipped 1 previous similar message [14338.016915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14338.017423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14338.032014] Lustre: Skipped 1 previous similar message [14340.545178] Lustre: server umount lustre-MDT0000 complete [14344.075272] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796023 with bad export cookie 7729237870701965446 [14344.080156] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14344.102034] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14344.422522] Lustre: server umount lustre-MDT0001 complete [14355.205301] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14358.369195] Key type lgssc unregistered [14358.650788] LNet: 468115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14358.656392] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14358.661574] LNet: Removed LNI 192.168.201.146@tcp [14359.711251] LNet: Removed LNI 192.168.201.146@tcp100 [14359.712932] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14359.718223] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14359.725251] LNet: Removed LNI 192.168.201.146@tcp101 [14360.735207] LNet: Removed LNI 192.168.201.146@tcp102 [14360.751777] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14360.769948] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14360.787274] LNet: Removed LNI 192.168.201.146@tcp103 [14361.823224] LNet: Removed LNI 192.168.201.146@tcp104 [14362.848660] LNet: Removed LNI 192.168.201.146@tcp105 [14362.852788] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14362.864011] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14362.878702] LNet: Removed LNI 192.168.201.146@tcp106 [14363.935163] LNet: Removed LNI 192.168.201.146@tcp107 [14364.960823] LNet: Removed LNI 192.168.201.146@tcp108 [14365.983747] LNet: Removed LNI 192.168.201.146@tcp109 [14367.007200] LNet: Removed LNI 192.168.201.146@tcp110 [14367.010271] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14367.026430] LNetError: 468115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14367.044458] LNet: Removed LNI 192.168.201.146@tcp111 [14368.096899] LNet: Removed LNI 192.168.201.146@tcp112 [14369.119408] LNet: Removed LNI 192.168.201.146@tcp113 [14370.144067] LNet: Removed LNI 192.168.201.146@tcp114 [14371.167248] LNet: Removed LNI 192.168.201.146@tcp115 [14372.191633] LNet: Removed LNI 192.168.201.146@tcp116 [14373.215285] LNet: Removed LNI 192.168.201.146@tcp117 [14374.239328] LNet: Removed LNI 192.168.201.146@tcp118 [14374.244374] LNet: Removed LNI 192.168.201.146@tcp119 [14374.249124] LNet: Removed LNI 192.168.201.146@tcp120 [14374.251102] LNet: Removed LNI 192.168.201.146@tcp121 [14374.266449] LNet: Removed LNI 192.168.201.146@tcp122 [14374.279235] LNet: Removed LNI 192.168.201.146@tcp123 [14374.284355] LNet: Removed LNI 192.168.201.146@tcp124 [14374.288569] LNet: Removed LNI 192.168.201.146@tcp125 [14374.294453] LNet: Removed LNI 192.168.201.146@tcp126 [14374.300445] LNet: Removed LNI 192.168.201.146@tcp127 [14374.306641] LNet: Removed LNI 192.168.201.146@tcp128 [14374.310632] LNet: Removed LNI 192.168.201.146@tcp129 [14374.320192] LNet: Removed LNI 192.168.201.146@tcp130 [14374.324838] LNet: Removed LNI 192.168.201.146@tcp131 [14374.329153] LNet: Removed LNI 192.168.201.146@tcp132 [14374.332729] LNet: Removed LNI 192.168.201.146@tcp133 [14374.349644] LNet: Removed LNI 192.168.201.146@tcp134 [14374.368525] LNet: Removed LNI 192.168.201.146@tcp135 [14375.225502] Key type .llcrypt unregistered [14375.227783] Key type ._llcrypt unregistered [14400.494590] Key type ._llcrypt registered [14400.497755] Key type .llcrypt registered [14400.652582] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [14414.918433] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14416.146329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14416.208333] alg: No test for adler32 (adler32-zlib) [14417.267778] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [14417.506667] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14419.183224] Key type lgssc registered [14420.520585] Lustre: Echo OBD driver; http://www.lustre.org/ [14458.031989] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:02:15 (1787796135) [14479.581170] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [14482.676246] Key type lgssc unregistered [14483.083650] LNet: 472217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14483.089702] LNetError: 472217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14483.104637] LNet: Removed LNI 192.168.201.146@tcp [14484.097596] Key type .llcrypt unregistered [14484.099357] Key type ._llcrypt unregistered [14502.911707] Key type ._llcrypt registered [14502.914547] Key type .llcrypt registered [14503.098071] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14504.325328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14504.375453] alg: No test for adler32 (adler32-zlib) [14505.461655] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [14505.711548] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [14507.400467] Key type lgssc registered [14508.757060] Lustre: Echo OBD driver; http://www.lustre.org/ [14518.262669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14518.288934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14519.656221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14519.694782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14519.815475] Lustre: lustre-MDT0000: new disk, initializing [14519.955254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14519.995658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14524.690932] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14534.488401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14534.649177] Lustre: 473435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14534.735822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14534.739332] Lustre: Skipped 1 previous similar message [14534.812311] Lustre: lustre-MDT0001: new disk, initializing [14534.894628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14534.917266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14534.925222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14539.350612] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14548.615395] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14554.749810] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14562.045552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14562.442595] Lustre: lustre-OST0000: new disk, initializing [14562.445753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14562.451978] Lustre: 475405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14562.547781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14568.096615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14568.116630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14568.267897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14569.848459] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14581.034871] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14586.120464] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing lnet_if_list [14592.829269] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14592.858692] LNet: Added LNI 192.168.201.146@tcp10 [8/256/0/180] [14598.949859] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14598.984788] LNet: Added LNI 192.168.201.146@tcp11 [8/256/0/180] [14604.241433] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14604.316648] LNet: Added LNI 192.168.201.146@tcp12 [8/256/0/180] [14609.318782] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14609.339731] LNet: Added LNI 192.168.201.146@tcp13 [8/256/0/180] [14613.989643] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14614.036939] LNet: Added LNI 192.168.201.146@tcp14 [8/256/0/180] [14619.120912] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14619.155237] LNet: Added LNI 192.168.201.146@tcp15 [8/256/0/180] [14624.043978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14624.082217] LNet: Added LNI 192.168.201.146@tcp16 [8/256/0/180] [14628.277663] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14628.304714] LNet: Added LNI 192.168.201.146@tcp17 [8/256/0/180] [14633.007982] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14633.029050] LNet: Added LNI 192.168.201.146@tcp18 [8/256/0/180] [14637.846457] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14637.873492] LNet: Added LNI 192.168.201.146@tcp19 [8/256/0/180] [14644.039874] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14644.069330] LNet: Added LNI 192.168.201.146@tcp20 [8/256/0/180] [14649.301916] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14649.341268] LNet: Added LNI 192.168.201.146@tcp21 [8/256/0/180] [14655.357526] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14655.406271] LNet: Added LNI 192.168.201.146@tcp22 [8/256/0/180] [14661.543089] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14661.568596] LNet: Added LNI 192.168.201.146@tcp23 [8/256/0/180] [14668.293515] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14668.314936] LNet: Added LNI 192.168.201.146@tcp24 [8/256/0/180] [14674.000699] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14674.025076] LNet: Added LNI 192.168.201.146@tcp25 [8/256/0/180] [14680.080654] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14680.130843] LNet: Added LNI 192.168.201.146@tcp26 [8/256/0/180] [14686.961140] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14687.002450] LNet: Added LNI 192.168.201.146@tcp27 [8/256/0/180] [14693.082600] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14693.137498] LNet: Added LNI 192.168.201.146@tcp28 [8/256/0/180] [14699.476701] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14699.501570] LNet: Added LNI 192.168.201.146@tcp29 [8/256/0/180] [14705.274165] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14705.297041] LNet: Added LNI 192.168.201.146@tcp30 [8/256/0/180] [14712.066316] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14712.101217] LNet: Added LNI 192.168.201.146@tcp31 [8/256/0/180] [14718.757589] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14718.786624] LNet: Added LNI 192.168.201.146@tcp32 [8/256/0/180] [14725.623917] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14725.647456] LNet: Added LNI 192.168.201.146@tcp33 [8/256/0/180] [14731.936450] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14731.962545] LNet: Added LNI 192.168.201.146@tcp34 [8/256/0/180] [14739.902603] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14739.937747] LNet: Added LNI 192.168.201.146@tcp35 [8/256/0/180] [14747.978448] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14748.017311] LNet: Added LNI 192.168.201.146@tcp36 [8/256/0/180] [14755.824503] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14755.844540] LNet: Added LNI 192.168.201.146@tcp37 [8/256/0/180] [14762.588656] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14762.607294] LNet: Added LNI 192.168.201.146@tcp38 [8/256/0/180] [14769.723748] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14769.749204] LNet: Added LNI 192.168.201.146@tcp39 [8/256/0/180] [14776.828978] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14776.865430] LNet: Added LNI 192.168.201.146@tcp40 [8/256/0/180] [14784.638787] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14784.684795] LNet: Added LNI 192.168.201.146@tcp41 [8/256/0/180] [14794.258339] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14794.282849] LNet: Added LNI 192.168.201.146@tcp42 [8/256/0/180] [14802.307395] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14802.332796] LNet: Added LNI 192.168.201.146@tcp43 [8/256/0/180] [14811.688145] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14811.721491] LNet: Added LNI 192.168.201.146@tcp44 [8/256/0/180] [14821.732510] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14821.751114] LNet: Added LNI 192.168.201.146@tcp45 [8/256/0/180] [14830.105277] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14830.137923] LNet: Added LNI 192.168.201.146@tcp46 [8/256/0/180] [14838.771300] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14838.874266] LNet: Added LNI 192.168.201.146@tcp47 [8/256/0/180] [14847.659897] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14847.679384] LNet: Added LNI 192.168.201.146@tcp48 [8/256/0/180] [14857.676733] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14857.702946] LNet: Added LNI 192.168.201.146@tcp49 [8/256/0/180] [14866.785812] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14866.810434] LNet: Added LNI 192.168.201.146@tcp50 [8/256/0/180] [14885.860316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14885.864908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14885.876463] Lustre: Skipped 1 previous similar message [14885.905853] Lustre: Skipped 2 previous similar messages [14890.999802] Lustre: server umount lustre-MDT0000 complete [14894.842729] LustreError: 473427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796574 with bad export cookie 11552692161123687119 [14894.851462] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14894.856743] LustreError: 473427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14895.270447] Lustre: server umount lustre-MDT0001 complete [14911.455150] Lustre: 472707:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796575/real 1787796575] req@00000000ff921061 x1874640172649728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787796591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14911.489856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14911.518553] Lustre: Skipped 1 previous similar message [14914.448320] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [14914.591464] Lustre: 472707:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796578/real 1787796578] req@00000000074e6b80 x1874640172649856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787796594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14917.602065] Lustre: 472708:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796581/real 1787796581] req@00000000149b4669 x1874640172649984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787796597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14920.682550] Lustre: 472709:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796584/real 1787796584] req@00000000acb85912 x1874640172650240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787796600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14922.435865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14925.791339] Lustre: 472706:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796589/real 1787796589] req@00000000009564ff x1874640172650368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787796605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14939.168902] LustreError: 472705:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059a2e900 x1874640172651136/t0(0) o250->MGC192.168.201.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14939.371577] LustreError: 475397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14939.386141] LustreError: 475397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14939.421258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14940.464947] LustreError: 475398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14940.470233] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14942.498719] LustreError: 475397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14943.731277] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14947.824836] LustreError: 475398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14952.315072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14952.688926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14956.785203] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14956.786114] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [14964.354508] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14970.652589] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14990.816940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14990.820346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14990.856090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14992.868419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14992.870213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14995.251225] Lustre: server umount lustre-OST0000 complete [15003.104872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15003.134103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15003.150675] Lustre: Skipped 2 previous similar messages [15005.551499] Lustre: server umount lustre-MDT0000 complete [15008.227134] LustreError: 499647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15008.260924] LustreError: 499647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15009.719249] LustreError: 473426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796689 with bad export cookie 11552692161123689695 [15009.738964] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15009.748419] LustreError: 473426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15010.294863] Lustre: server umount lustre-MDT0001 complete [15024.850054] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15029.672372] Key type lgssc unregistered [15030.189441] LNet: 502445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15030.211166] LNetError: 502445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15030.224968] LNet: Removed LNI 192.168.201.146@tcp [15030.230765] LNet: Removed LNI 192.168.201.146@tcp10 [15030.238293] LNet: Removed LNI 192.168.201.146@tcp11 [15030.241792] LNet: Removed LNI 192.168.201.146@tcp12 [15030.250481] LNet: Removed LNI 192.168.201.146@tcp13 [15030.253759] LNet: Removed LNI 192.168.201.146@tcp14 [15030.258407] LNet: Removed LNI 192.168.201.146@tcp15 [15030.267579] LNet: Removed LNI 192.168.201.146@tcp16 [15030.273369] LNet: Removed LNI 192.168.201.146@tcp17 [15030.277766] LNet: Removed LNI 192.168.201.146@tcp18 [15030.283945] LNet: Removed LNI 192.168.201.146@tcp19 [15030.297574] LNet: Removed LNI 192.168.201.146@tcp20 [15030.303456] LNet: Removed LNI 192.168.201.146@tcp21 [15030.309750] LNet: Removed LNI 192.168.201.146@tcp22 [15030.318763] LNet: Removed LNI 192.168.201.146@tcp23 [15030.322768] LNet: Removed LNI 192.168.201.146@tcp24 [15030.328500] LNet: Removed LNI 192.168.201.146@tcp25 [15030.333676] LNet: Removed LNI 192.168.201.146@tcp26 [15030.343228] LNet: Removed LNI 192.168.201.146@tcp27 [15030.357779] LNet: Removed LNI 192.168.201.146@tcp28 [15030.364320] LNet: Removed LNI 192.168.201.146@tcp29 [15030.373053] LNet: Removed LNI 192.168.201.146@tcp30 [15030.375730] LNet: Removed LNI 192.168.201.146@tcp31 [15030.383731] LNet: Removed LNI 192.168.201.146@tcp32 [15030.385567] LNet: Removed LNI 192.168.201.146@tcp33 [15030.399254] LNet: Removed LNI 192.168.201.146@tcp34 [15030.404984] LNet: Removed LNI 192.168.201.146@tcp35 [15030.409797] LNet: Removed LNI 192.168.201.146@tcp36 [15030.412453] LNet: Removed LNI 192.168.201.146@tcp37 [15030.416249] LNet: Removed LNI 192.168.201.146@tcp38 [15030.419880] LNet: Removed LNI 192.168.201.146@tcp39 [15030.423235] LNet: Removed LNI 192.168.201.146@tcp40 [15030.425972] LNet: Removed LNI 192.168.201.146@tcp41 [15030.428755] LNet: Removed LNI 192.168.201.146@tcp42 [15030.431836] LNet: Removed LNI 192.168.201.146@tcp43 [15030.434550] LNet: Removed LNI 192.168.201.146@tcp44 [15030.437653] LNet: Removed LNI 192.168.201.146@tcp45 [15030.440802] LNet: Removed LNI 192.168.201.146@tcp46 [15030.442849] LNet: Removed LNI 192.168.201.146@tcp47 [15030.447186] LNet: Removed LNI 192.168.201.146@tcp48 [15030.450869] LNet: Removed LNI 192.168.201.146@tcp49 [15030.483476] LNet: Removed LNI 192.168.201.146@tcp50 [15031.899252] Key type .llcrypt unregistered [15031.900898] Key type ._llcrypt unregistered [15051.561150] Key type ._llcrypt registered [15051.565106] Key type .llcrypt registered [15051.711901] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15065.283600] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15066.385862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15066.465281] alg: No test for adler32 (adler32-zlib) [15067.668544] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [15068.007501] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15069.695767] Key type lgssc registered [15070.968804] Lustre: Echo OBD driver; http://www.lustre.org/ [15100.725914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15100.743908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15101.980333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15102.004440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15102.065139] Lustre: lustre-MDT0000: new disk, initializing [15102.135079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15102.157738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15105.984526] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15116.612832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15116.810119] Lustre: 505635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15116.884720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15116.888695] Lustre: Skipped 1 previous similar message [15116.999367] Lustre: lustre-MDT0001: new disk, initializing [15117.098532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15117.171903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15117.198697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15121.505716] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15132.414786] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15138.987052] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15147.793837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15148.129122] Lustre: *** cfs_fail_loc=90f, val=0*** [15148.299054] Lustre: lustre-OST0000: new disk, initializing [15148.304459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15148.315958] Lustre: 507651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15148.500973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15153.916705] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15157.854223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15157.868190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15157.999274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15162.141721] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15173.600599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15173.604803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15173.619980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15173.623248] Lustre: Skipped 1 previous similar message [15175.132542] Lustre: server umount lustre-OST0000 complete [15183.840167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15183.850825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15183.869088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15183.871498] Lustre: Skipped 2 previous similar messages [15185.647459] Lustre: server umount lustre-MDT0000 complete [15189.090108] LustreError: 505628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796868 with bad export cookie 3506912814588003089 [15189.102207] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15189.106914] LustreError: 505628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15189.455890] Lustre: server umount lustre-MDT0001 complete [15199.603605] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15201.978908] Key type lgssc unregistered [15202.316279] LNet: 509558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15202.320734] LNetError: 509558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15202.343468] LNet: Removed LNI 192.168.201.146@tcp [15203.107259] Key type .llcrypt unregistered [15203.108608] Key type ._llcrypt unregistered [15219.239957] Key type ._llcrypt registered [15219.242153] Key type .llcrypt registered [15219.330124] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15220.282235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15220.339871] alg: No test for adler32 (adler32-zlib) [15221.513107] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [15221.764886] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15223.431167] Key type lgssc registered [15224.326746] Lustre: Echo OBD driver; http://www.lustre.org/ [15230.915338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15230.931770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15232.373546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15236.815157] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15242.727253] LustreError: 510810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15242.766723] LustreError: 510810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15245.570435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15246.119513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15250.220760] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15259.461489] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15265.465731] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15272.634384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15273.085515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15279.875163] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15287.109063] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15300.135653] loop0: detected capacity change from 0 to 204800000 [15300.326816] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15300.339802] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15300.351837] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15306.161944] loop0: detected capacity change from 0 to 204800000 [15306.358840] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15306.365506] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15306.374846] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15311.904266] loop0: detected capacity change from 0 to 204800000 [15317.418700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15317.627572] Lustre: fs15246-OST0000: new disk, initializing [15317.631586] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15317.637968] Lustre: 514400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15317.699380] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15323.886615] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15333.644314] loop1: detected capacity change from 0 to 204800000 [15339.492367] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15339.610363] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15339.659211] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15339.736774] Lustre: fs15246-MDT0000: new disk, initializing [15339.929517] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15339.950721] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15340.090154] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15341.152238] Lustre: 515521:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a8ce0cc5 x1874640923310208/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787797037 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15343.982351] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15346.217539] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15346.231515] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15346.264923] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15361.524585] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15361.535060] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15366.631823] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15367.153919] Lustre: server umount fs15246-MDT0000 complete [15371.751667] LustreError: 510805:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15375.510941] Lustre: server umount fs15246-OST0000 complete [15387.114896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15387.126206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15387.132123] Lustre: Skipped 1 previous similar message [15392.224182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15392.235098] Lustre: Skipped 1 previous similar message [15392.667568] Lustre: server umount lustre-OST0000 complete [15397.345786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15397.352319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15397.366712] Lustre: Skipped 2 previous similar messages [15402.714445] Lustre: server umount lustre-MDT0000 complete [15406.202771] LustreError: 510791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797086 with bad export cookie 9329079856853453528 [15406.220295] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15406.629412] Lustre: server umount lustre-MDT0001 complete [15416.771366] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15420.313554] Key type lgssc unregistered [15420.601801] LNet: 518620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15420.631418] LNetError: 518620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15420.678581] LNet: Removed LNI 192.168.201.146@tcp [15421.884856] Key type .llcrypt unregistered [15421.886245] Key type ._llcrypt unregistered [15445.388302] Key type ._llcrypt registered [15445.398537] Key type .llcrypt registered [15445.514994] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15446.870523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15446.882276] alg: No test for adler32 (adler32-zlib) [15448.671709] Lustre: Lustre: Build Version: 2.17.57_56_gf34a808 [15448.910801] LNet: Added LNI 192.168.201.146@tcp [8/256/0/180] [15450.607291] Key type lgssc registered [15451.763283] Lustre: Echo OBD driver; http://www.lustre.org/ [15496.464812] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15507.374743] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15546.530168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15546.576091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15547.877730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15547.907368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15548.000443] Lustre: lustre-MDT0000: new disk, initializing [15548.131737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15548.161808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15552.613645] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15563.078142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15563.283291] Lustre: 523086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15563.376946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15563.381163] Lustre: Skipped 1 previous similar message [15563.438401] Lustre: lustre-MDT0001: new disk, initializing [15563.513675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15563.549054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15563.567316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15568.235217] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15579.357738] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15585.606588] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15592.835632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15593.073207] Lustre: lustre-OST0000: new disk, initializing [15593.078108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15593.083970] Lustre: 525056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15593.161598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15599.713485] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_default_debug -1 all [15602.224482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15602.240739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15602.312778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15608.881958] Lustre: DEBUG MARKER: oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15613.523489] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15613.719375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15618.906846] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15619.159105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15622.636720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15622.657831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15622.667910] Lustre: Skipped 1 previous similar message [15627.490616] Lustre: server umount lustre-OST0000 complete [15632.868147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15632.873075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15632.888177] Lustre: Skipped 2 previous similar messages [15638.236540] Lustre: server umount lustre-MDT0000 complete [15643.106417] LustreError: 523094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15643.117969] LustreError: 523094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15643.275323] LustreError: 523080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797323 with bad export cookie 7430294614034792912 [15643.287546] LustreError: MGC192.168.201.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15643.291126] LustreError: 523080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15643.690607] Lustre: server umount lustre-MDT0001 complete [15670.207739] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing set_hostid [15681.094184] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing load_modules_local [15715.161940] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15429 sec ========= 22:23:12 (1787797392) [15716.785519] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:23:14 (1787797394) === [15719.799536] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:23:17 (1787797397) === [15756.452807] Lustre: DEBUG MARKER: oleg146-server.virtnet: executing unload_modules_local [15758.870735] Key type lgssc unregistered [15759.130827] LNet: 531704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15759.143316] LNetError: 531704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15759.161976] LNet: Removed LNI 192.168.201.146@tcp [15759.953342] Key type .llcrypt unregistered [15759.956123] Key type ._llcrypt unregistered