[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507555764 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003125] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012074] Yama: becoming mindful. [ 0.013050] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022805] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027383] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031505] *** VALIDATE proc *** [ 0.032255] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041398] debug: unmapping init [mem 0xffffffff95059000-0xffffffff95060fff] [ 0.043174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052355] rcu: Hierarchical SRCU implementation. [ 0.054465] smp: Bringing up secondary CPUs ... [ 0.055608] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.060085] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139000] node 0 deferred pages initialised in 73ms [ 0.142147] devtmpfs: initialized [ 0.143215] x86/mm: Memory block size: 128MB [ 0.145688] gcov: version magic: 0x41383552 [ 0.147293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149248] pinctrl core: initialized pinctrl subsystem [ 0.150181] [ 0.151009] ************************************************************* [ 0.152010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153010] ** ** [ 0.154010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155011] ** ** [ 0.156010] ** This means that this kernel is built to expose internal ** [ 0.157010] ** IOMMU data structures, which may compromise security on ** [ 0.158010] ** your system. ** [ 0.159009] ** ** [ 0.160008] ** If you see this message and you are not debugging the ** [ 0.161007] ** kernel, report this immediately to your vendor! ** [ 0.162011] ** ** [ 0.163010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164011] ************************************************************* [ 0.165639] NET: Registered protocol family 16 [ 0.166437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169498] cpuidle: using governor menu [ 0.171801] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174485] PCI: Using configuration type 1 for base access [ 0.177125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192109] cryptd: max_cpu_qlen set to 1000 [ 0.195238] ACPI: Added _OSI(Module Device) [ 0.196008] ACPI: Added _OSI(Processor Device) [ 0.197007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197901] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200072] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203204] ACPI: Interpreter enabled [ 0.204056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205009] ACPI: Using IOAPIC for interrupt routing [ 0.206095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.217045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.219078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221379] acpiphp: Slot [2] registered [ 0.222107] acpiphp: Slot [5] registered [ 0.223109] acpiphp: Slot [6] registered [ 0.224114] acpiphp: Slot [7] registered [ 0.225100] acpiphp: Slot [8] registered [ 0.226147] acpiphp: Slot [9] registered [ 0.227132] acpiphp: Slot [10] registered [ 0.228121] acpiphp: Slot [3] registered [ 0.229111] acpiphp: Slot [4] registered [ 0.231114] acpiphp: Slot [11] registered [ 0.233103] acpiphp: Slot [12] registered [ 0.234111] acpiphp: Slot [13] registered [ 0.236108] acpiphp: Slot [14] registered [ 0.237091] acpiphp: Slot [15] registered [ 0.238052] acpiphp: Slot [16] registered [ 0.240119] acpiphp: Slot [17] registered [ 0.242109] acpiphp: Slot [18] registered [ 0.243096] acpiphp: Slot [19] registered [ 0.244087] acpiphp: Slot [20] registered [ 0.245095] acpiphp: Slot [21] registered [ 0.246093] acpiphp: Slot [22] registered [ 0.248122] acpiphp: Slot [23] registered [ 0.249113] acpiphp: Slot [24] registered [ 0.251096] acpiphp: Slot [25] registered [ 0.252086] acpiphp: Slot [26] registered [ 0.254101] acpiphp: Slot [27] registered [ 0.255105] acpiphp: Slot [28] registered [ 0.257099] acpiphp: Slot [29] registered [ 0.258087] acpiphp: Slot [30] registered [ 0.260091] acpiphp: Slot [31] registered [ 0.261106] PCI host bridge to bus 0000:00 [ 0.263030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.267029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270076] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278199] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281110] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285266] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295839] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299952] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.302018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.307018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310521] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.312702] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317802] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349997] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356749] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.376016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.393018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482282] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525027] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565805] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.574015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.597017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.608000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.616018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.623018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.652906] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.655343] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661271] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667024] iommu: Default domain type: Passthrough [ 0.668436] SCSI subsystem initialized [ 0.669121] ACPI: bus type USB registered [ 0.671140] usbcore: registered new interface driver usbfs [ 0.673075] usbcore: registered new interface driver hub [ 0.675077] usbcore: registered new device driver usb [ 0.677175] pps_core: LinuxPPS API ver. 1 registered [ 0.679012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.683065] PTP clock support registered [ 0.685074] EDAC MC: Ver: 3.0.0 [ 0.689116] PCI: Using ACPI for IRQ routing [ 0.691867] NetLabel: Initializing [ 0.693011] NetLabel: domain hash size = 128 [ 0.694009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.697076] NetLabel: unlabeled traffic allowed by default [ 0.699126] vgaarb: loaded [ 0.700241] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.702014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.711000] clocksource: Switched to clocksource kvm-clock [ 0.824790] VFS: Disk quotas dquot_6.6.0 [ 0.826236] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.828652] *** VALIDATE ramfs *** [ 0.829834] *** VALIDATE hugetlbfs *** [ 0.831321] pnp: PnP ACPI init [ 0.833664] pnp: PnP ACPI: found 6 devices [ 0.850889] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854129] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.856334] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858402] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860682] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863372] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866531] NET: Registered protocol family 2 [ 0.869638] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874631] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.878234] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.883664] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.886963] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889550] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892098] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894554] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897376] NET: Registered protocol family 1 [ 0.900296] RPC: Registered named UNIX socket transport module. [ 0.902864] RPC: Registered udp transport module. [ 0.904703] RPC: Registered tcp transport module. [ 0.906475] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.908818] NET: Registered protocol family 44 [ 0.910508] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.912518] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.914529] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.916823] PCI: CLS 0 bytes, default 64 [ 0.918378] Unpacking initramfs... [ 2.272090] debug: unmapping init [mem 0xffff9ef2bcc54000-0xffff9ef2bffbffff] [ 2.276137] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.278477] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.281620] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.776876] Initialise system trusted keyrings [ 2.778789] Key type blacklist registered [ 2.781243] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.790724] zbud: loaded [ 2.795367] *** VALIDATE nfs *** [ 2.796717] *** VALIDATE nfs4 *** [ 2.798398] pstore: using deflate compression [ 2.801476] Platform Keyring initialized [ 2.899945] NET: Registered protocol family 38 [ 2.901476] Key type asymmetric registered [ 2.902653] Asymmetric key parser 'x509' registered [ 2.904063] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.906344] io scheduler mq-deadline registered [ 2.908194] io scheduler kyber registered [ 2.909942] io scheduler bfq registered [ 2.912179] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.915246] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.918231] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.921058] ACPI: Power Button [PWRF] [ 2.926097] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.932669] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.947209] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.957115] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.983170] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.010178] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.039646] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.044501] Non-volatile memory driver v1.3 [ 3.046030] Linux agpgart interface v0.103 [ 3.080828] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.083201] vda: detected capacity change from 0 to 74702848 [ 3.098451] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.101399] vdb: detected capacity change from 0 to 1073741824 [ 3.118127] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.121026] vdc: detected capacity change from 0 to 2621440000 [ 3.136659] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.139266] vdd: detected capacity change from 0 to 2621440000 [ 3.154281] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.157214] vde: detected capacity change from 0 to 4294967296 [ 3.172890] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.175692] vdf: detected capacity change from 0 to 4294967296 [ 3.181402] libphy: Fixed MDIO Bus: probed [ 3.186162] usbcore: registered new interface driver usbserial_generic [ 3.188680] usbserial: USB Serial support registered for generic [ 3.190946] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.193817] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.194892] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.197137] mousedev: PS/2 mouse device common for all mice [ 3.199511] rtc_cmos 00:05: RTC can wake from S4 [ 3.201817] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202876] rtc_cmos 00:05: registered as rtc0 [ 3.206773] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.207749] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.211558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.212819] intel_pstate: CPU model not supported [ 3.217879] hid: raw HID events driver (C) Jiri Kosina [ 3.219840] usbcore: registered new interface driver usbhid [ 3.221759] usbhid: USB HID core driver [ 3.223853] drop_monitor: Initializing network drop monitor service [ 3.225856] Initializing XFRM netlink socket [ 3.227797] NET: Registered protocol family 10 [ 3.230549] Segment Routing with IPv6 [ 3.231531] NET: Registered protocol family 17 [ 3.233047] mpls_gso: MPLS GSO support [ 3.239147] RAS: Correctable Errors collector initialized. [ 3.241464] AVX version of gcm_enc/dec engaged. [ 3.243067] AES CTR mode by8 optimization enabled [ 3.313254] sched_clock: Marking stable (3313231636, 0)->(4224192948, -910961312) [ 3.316497] registered taskstats version 1 [ 3.318143] Loading compiled-in X.509 certificates [ 3.319857] zswap: loaded using pool lzo/zbud [ 3.342898] Key type big_key registered [ 3.354815] Key type encrypted registered [ 3.356412] ima: No TPM chip found, activating TPM-bypass! [ 3.358059] ima: Allocated hash algorithm: sha1 [ 3.359536] ima: No architecture policies found [ 3.361238] evm: Initialising EVM extended attributes: [ 3.362666] evm: security.selinux [ 3.363670] evm: security.ima [ 3.364584] evm: security.capability [ 3.365608] evm: HMAC attrs: 0x1 [ 3.367679] rtc_cmos 00:05: setting system clock to 2026-08-09 21:18:47 UTC (1786310327) [ 3.373432] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.375980] debug: unmapping init [mem 0xffffffff94d82000-0xffffffff95058fff] [ 3.383092] Write protecting the kernel read-only data: 28672k [ 3.386896] debug: unmapping init [mem 0xffffffff93403000-0xffffffff935fffff] [ 3.389133] debug: unmapping init [mem 0xffffffff93d14000-0xffffffff93dfffff] [ 3.418356] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.425375] systemd[1]: Detected virtualization kvm. [ 3.427144] systemd[1]: Detected architecture x86-64. [ 3.428647] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.450022] systemd[1]: No hostname configured. [ 3.451374] systemd[1]: Set hostname to . [ 3.452969] random: systemd: uninitialized urandom read (16 bytes read) [ 3.455583] systemd[1]: Initializing machine ID from random generator. [ 3.497997] random: ln: uninitialized urandom read (6 bytes read) [ 3.583095] random: systemd: uninitialized urandom read (16 bytes read) [ 3.586257] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.591321] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.600151] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.115830] device-mapper: uevent: version 1.0.3 [ 4.117881] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.737547] virtio_net virtio0 ens2: renamed from eth0 [ 4.740604] random: fast init done [ 4.772612] scsi host0: ata_piix [ 4.791197] scsi host1: ata_piix [ 4.792864] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.794962] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.122931] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.871577] random: crng init done [ 9.873284] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.148704] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.210564] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.438169] SELinux: Disabled at runtime. [ 11.496537] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.504909] systemd[1]: Detected virtualization kvm. [ 11.506766] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.999612] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.002842] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.011071] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.014974] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.018631] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.029532] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.037456] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ 12.163067] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.571992] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.905954] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.980102] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.160316] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.186497] EDAC sbridge: Ver: 1.1.2 [ 14.826436] Key type dns_resolver registered [ 15.152561] NFS: Registering the id_resolver key type [ 15.154780] Key type id_resolver registered [ 15.156671] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg211-server login: [ 41.184653] libcfs: loading out-of-tree module taints kernel. [ 41.206181] Key type ._llcrypt registered [ 41.207746] Key type .llcrypt registered [ 41.262888] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 49.674660] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 50.535645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.544958] alg: No test for adler32 (adler32-zlib) [ 51.593168] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 51.977362] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 53.607213] Key type lgssc registered [ 55.729962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.430611] hrtimer: interrupt took 2402990 ns [ 74.707827] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.590744] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 132.842839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.887175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.189480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.225557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.348169] Lustre: lustre-MDT0000: new disk, initializing [ 134.422212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.444276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.721620] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.479530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.593885] Lustre: 6490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 151.623882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 151.627876] Lustre: Skipped 1 previous similar message [ 151.719344] Lustre: lustre-MDT0001: new disk, initializing [ 151.797201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 151.836544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 151.849731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.116366] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.328245] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.509452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.964140] Lustre: lustre-OST0000: new disk, initializing [ 168.968589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.975336] Lustre: 8426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 169.053341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.603062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 174.616080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.696740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 174.924640] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.859939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.017684] Lustre: lustre-OST0001: new disk, initializing [ 190.021293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.023802] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.095456] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.033961] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.698171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.702872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.751652] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.137957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.548389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 221.477627] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing check_logdir /tmp/testlogs/ [ 226.678873] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing yml_node [ 230.478521] Lustre: DEBUG MARKER: Client: 2.17.56.55 [ 232.724434] Lustre: DEBUG MARKER: MDS: 2.17.56.55 [ 234.802913] Lustre: DEBUG MARKER: OSS: 2.17.56.55 [ 235.968348] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 17:22:39 EDT 2026 [ 250.638302] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 251.931951] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 269.280190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.292236] Lustre: Skipped 2 previous similar messages [ 269.299920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.311236] Lustre: Skipped 3 previous similar messages [ 273.655311] Lustre: server umount lustre-MDT0000 complete [ 277.778041] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310601 with bad export cookie 1084167485453688243 [ 277.789097] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.800512] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 279.525081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.528063] LustreError: 12240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.528225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.543239] Lustre: Skipped 1 previous similar message [ 279.581105] LustreError: 12240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 282.528090] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.528719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.558518] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 282.588481] Lustre: Skipped 2 previous similar messages [ 287.649422] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.649555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.666166] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 287.696222] Lustre: Skipped 1 previous similar message [ 292.319674] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 292.752737] Lustre: server umount lustre-MDT0001 complete [ 297.606787] Lustre: server umount lustre-OST0000 complete [ 302.185465] Lustre: server umount lustre-OST0001 complete [ 308.561811] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 319.185950] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 352.561892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.749148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.778280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.849311] Lustre: lustre-MDT0000: new disk, initializing [ 352.914205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.934956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 357.112191] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 367.986237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.141597] Lustre: 15054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 368.210463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 368.214665] Lustre: Skipped 1 previous similar message [ 368.358298] Lustre: lustre-MDT0001: new disk, initializing [ 368.497737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 368.555249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 368.574053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.074750] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 381.864682] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 386.994462] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 393.476314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 393.852656] Lustre: lustre-OST0000: new disk, initializing [ 393.856308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 393.861337] Lustre: 17026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 393.960981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.199248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 395.214304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 395.337995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.073401] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 409.759828] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 414.001987] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 414.217742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 417.979919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 418.135977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 420.835500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.839185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.846029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.858735] Lustre: Skipped 1 previous similar message [ 420.877041] Lustre: Skipped 1 previous similar message [ 425.738776] Lustre: server umount lustre-OST0000 complete [ 430.055816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 430.072134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.087477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 436.200403] LustreError: 15934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.217375] LustreError: 15934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 436.249593] Lustre: server umount lustre-MDT0000 complete [ 439.957730] LustreError: 17037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310764 with bad export cookie 1084167485453689776 [ 439.958994] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 439.967073] LustreError: 17037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 440.239639] Lustre: server umount lustre-MDT0001 complete [ 446.000608] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:26:08 (1786310768) [ 465.029992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.610277] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.695345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.887319] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 475.111292] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.152354] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 480.643201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.706363] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 494.586153] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.263717] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.468751] Lustre: Failing over lustre-MDT0000 [ 502.832508] Lustre: server umount lustre-MDT0000 complete [ 506.611043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.625675] LustreError: 20855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.626649] Lustre: Skipped 2 previous similar messages [ 506.655680] LustreError: 20855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 522.210821] Lustre: 3623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310830/real 1786310830] req@ffff9ef206681180 x1873082188981504/t0(0) o400->MGC192.168.202.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786310846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 522.247272] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.969094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.814615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.819758] Lustre: Skipped 1 previous similar message [ 538.911770] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 539.120231] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 539.130666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 539.175063] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 549.092205] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 554.464725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 554.474787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.502710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.508206] Lustre: Skipped 2 previous similar messages [ 559.713196] Lustre: server umount lustre-MDT0000 complete [ 561.632846] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.661487] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 564.450906] LustreError: 19881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310888 with bad export cookie 1084167485453691190 [ 564.461460] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.469537] LustreError: 19881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 564.844922] Lustre: server umount lustre-MDT0001 complete [ 573.871596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.676072] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 590.317565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.747747] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 603.517873] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.387977] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.286759] Lustre: Failing over lustre-MDT0000 [ 616.311331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.312676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.316603] Lustre: Skipped 2 previous similar messages [ 616.322472] Lustre: Skipped 3 previous similar messages [ 616.585400] Lustre: server umount lustre-MDT0000 complete [ 626.662579] LustreError: 23240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.719153] LustreError: 23240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 636.456417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.611727] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.951498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 641.208120] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 642.028478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 642.036411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 642.065431] Lustre: Skipped 1 previous similar message [ 642.105995] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 654.789444] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 661.473164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.217686] Lustre: server umount lustre-MDT0000 complete [ 670.624165] LustreError: 23221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310994 with bad export cookie 1084167485453692128 [ 670.630124] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.911650] Lustre: server umount lustre-MDT0001 complete [ 681.608963] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:30:04 (1786311004) [ 700.462710] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 712.075396] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 750.327398] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 763.679770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.946157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 763.978210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 764.086832] Lustre: lustre-MDT0000: new disk, initializing [ 764.170822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.180589] Lustre: Skipped 3 previous similar messages [ 764.204121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 769.064444] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 780.718955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.846712] Lustre: 29618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 781.267819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 786.143439] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 797.597676] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 803.474400] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 810.526624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.941567] Lustre: 31586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 812.196796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 812.241184] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 812.397690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 812.424276] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 818.209814] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 827.650524] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 832.680693] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 853.500203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.510719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.525291] Lustre: Skipped 1 previous similar message [ 853.529640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.541949] Lustre: Skipped 4 previous similar messages [ 857.703936] Lustre: server umount lustre-OST0000 complete [ 863.205948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.223023] LustreError: Skipped 1 previous similar message [ 868.147501] Lustre: server umount lustre-MDT0000 complete [ 868.837374] LustreError: 30493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.879543] LustreError: 30493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 873.395388] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311197 with bad export cookie 1084167485453692534 [ 873.420634] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.435370] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 874.279367] Lustre: server umount lustre-MDT0001 complete [ 889.713499] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 893.455062] Key type lgssc unregistered [ 893.884306] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.890593] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.949758] LNet: Removed LNI 192.168.202.111@tcp [ 896.445493] Key type .llcrypt unregistered [ 896.452133] Key type ._llcrypt unregistered [ 925.318426] Key type ._llcrypt registered [ 925.321296] Key type .llcrypt registered [ 925.481428] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 927.291729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 927.374313] alg: No test for adler32 (adler32-zlib) [ 928.580611] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 928.921352] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 930.655735] Key type lgssc registered [ 932.414945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.193618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.215682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.784258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.535920] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 954.338817] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.352426] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 959.459662] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.859337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.379165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.268681] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 974.407448] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.531578] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 989.631865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.170809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.565985] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1005.589157] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1006.196195] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1014.915161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1031.512983] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:35:54 (1786311354) [ 1034.003162] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1035.915675] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1044.284891] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:36:06 (1786311366) [ 1055.308793] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:36:17 (1786311377) [ 1056.708708] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1068.707550] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1068.713748] Lustre: Skipped 2 previous similar messages [ 1071.405890] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:36:33 (1786311393) [ 1074.424984] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1074.441670] Lustre: Skipped 2 previous similar messages [ 1083.759952] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1083.766711] Lustre: Skipped 4 previous similar messages [ 1099.660664] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:37:02 (1786311422) [ 1102.859679] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1106.543884] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1115.740585] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:37:18 (1786311438) [ 1117.771487] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1117.782399] Lustre: Skipped 6 previous similar messages [ 1132.001483] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:37:34 (1786311454) [ 1153.009488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.013777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.023530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.066662] Lustre: Skipped 1 previous similar message [ 1156.097745] Lustre: server umount lustre-OST0000 complete [ 1163.235142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.238113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.250533] Lustre: Skipped 1 previous similar message [ 1163.276803] Lustre: Skipped 2 previous similar messages [ 1166.637437] Lustre: server umount lustre-MDT0000 complete [ 1168.352140] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.378556] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1171.010347] LustreError: 38569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311495 with bad export cookie 9061338830637221353 [ 1171.011357] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.018694] LustreError: 38569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1171.524833] Lustre: server umount lustre-MDT0001 complete [ 1184.273451] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1187.734786] Key type lgssc unregistered [ 1188.113855] LNet: 42872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.124959] LNetError: 42872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.147213] LNet: Removed LNI 192.168.202.111@tcp [ 1189.345492] Key type .llcrypt unregistered [ 1189.349550] Key type ._llcrypt unregistered [ 1210.349483] Key type ._llcrypt registered [ 1210.356324] Key type .llcrypt registered [ 1210.541099] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 1230.149295] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1231.568458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.724738] alg: No test for adler32 (adler32-zlib) [ 1233.097237] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1233.347415] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1235.063253] Key type lgssc registered [ 1236.174168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.416417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.431617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.681376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1273.711051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1273.764280] Lustre: lustre-MDT0000: new disk, initializing [ 1273.817124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.838397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1278.579699] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1289.840022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.009646] Lustre: 46026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1290.060980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1290.064584] Lustre: Skipped 1 previous similar message [ 1290.159324] Lustre: lustre-MDT0001: new disk, initializing [ 1290.378295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1290.457914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1290.471260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1295.629319] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1305.723361] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1311.253290] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1318.881346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.090583] Lustre: lustre-OST0000: new disk, initializing [ 1319.095144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1319.101388] Lustre: 47998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1319.197899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.671593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1325.689054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1325.897920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1326.536527] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1337.772790] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1343.741443] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1343.987923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1348.778907] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1348.970284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1351.654816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1351.663195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.685765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.694822] Lustre: Skipped 1 previous similar message [ 1356.772780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.789170] Lustre: Skipped 1 previous similar message [ 1357.525721] Lustre: server umount lustre-OST0000 complete [ 1367.012096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1367.015476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.030391] LustreError: Skipped 1 previous similar message [ 1367.056344] Lustre: Skipped 1 previous similar message [ 1367.062169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.672673] Lustre: server umount lustre-MDT0000 complete [ 1372.814059] LustreError: 46020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311696 with bad export cookie 7546416638406552810 [ 1372.824655] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.189029] Lustre: server umount lustre-MDT0001 complete [ 1383.754803] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:41:46 (1786311706) [ 1391.193406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.769148] LustreError: 50458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.790742] LustreError: 50458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1391.854740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.222527] LustreError: 50459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.398376] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1401.315478] LustreError: 50458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.451063] LustreError: 50459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.202408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.674328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.117648] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1422.335630] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1428.283571] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1435.764851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1436.230686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.237788] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1452.880782] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1457.611528] LustreError: 52986:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1458.634637] Lustre: Modifying parameter general.timeout=1 in log params [ 1459.718876] Lustre: Modifying parameter general.timeout=2 in log params [ 1461.926644] Lustre: Modifying parameter general.timeout=4 in log params [ 1461.937518] Lustre: Skipped 1 previous similar message [ 1466.375492] Lustre: Modifying parameter general.timeout=8 in log params [ 1466.383785] Lustre: Skipped 3 previous similar messages [ 1474.989769] Lustre: Modifying parameter general.timeout=16 in log params [ 1474.997804] Lustre: Skipped 7 previous similar messages [ 1491.620715] Lustre: Modifying parameter general.timeout=32 in log params [ 1491.626502] Lustre: Skipped 15 previous similar messages [ 1525.075406] Lustre: Modifying parameter general.timeout=63 in log params [ 1525.077966] Lustre: Skipped 30 previous similar messages [ 1597.175663] Lustre: Disabling parameter general.timeout= in log params [ 1597.182806] Lustre: Skipped 66 previous similar messages [ 1600.831738] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:45:23 (1786311923) [ 1604.147461] Lustre: 59936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1606.293307] Lustre: 60034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1608.396384] Lustre: 60145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1612.401503] Lustre: 60343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1612.409799] Lustre: 60343:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1645.682822] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1645.695943] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1645.741591] LustreError: 61818:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1651.364939] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:46:13 (1786311973) [ 1668.583167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.588927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.594054] Lustre: Skipped 1 previous similar message [ 1668.613328] Lustre: Skipped 3 previous similar messages [ 1672.307804] Lustre: server umount lustre-MDT0000 complete [ 1673.712059] LustreError: 51186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.726498] LustreError: 51186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1677.474853] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312001 with bad export cookie 7546416638406553692 [ 1677.478657] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.502948] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1678.581871] Lustre: server umount lustre-MDT0001 complete [ 1692.099417] Lustre: server umount lustre-OST0000 complete [ 1723.448288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.037446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1724.499138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.137746] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1739.814936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.997668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1740.092091] Lustre: 64354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1740.239777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1745.204092] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1752.448764] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.776422] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1766.546437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1766.859070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1767.345241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.349228] Lustre: Skipped 1 previous similar message [ 1774.782452] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1783.713732] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.732276] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1785.748814] Lustre: Skipped 23 previous similar messages [ 1785.801819] Lustre: 66921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1794.022915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.032081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.054266] Lustre: Skipped 1 previous similar message [ 1794.061416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.152774] Lustre: server umount lustre-OST0000 complete [ 1804.769415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.780163] Lustre: Skipped 1 previous similar message [ 1811.008471] Lustre: server umount lustre-MDT0000 complete [ 1815.010798] LustreError: 64364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.037491] LustreError: 64364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1815.400280] LustreError: 65094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312139 with bad export cookie 7546416638406556898 [ 1815.401864] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.418681] LustreError: 65094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1815.862323] Lustre: server umount lustre-MDT0001 complete [ 1829.928749] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1833.318369] Key type lgssc unregistered [ 1833.630765] LNet: 68134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.637799] LNetError: 68134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.651861] LNet: Removed LNI 192.168.202.111@tcp [ 1834.615450] Key type .llcrypt unregistered [ 1834.619494] Key type ._llcrypt unregistered [ 1864.092796] Key type ._llcrypt registered [ 1864.103308] Key type .llcrypt registered [ 1864.253710] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1865.524396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.567064] alg: No test for adler32 (adler32-zlib) [ 1866.837066] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1867.087220] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1868.848833] Key type lgssc registered [ 1870.335695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1879.959716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1880.002547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.746826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.257456] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1891.300644] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.315895] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1896.431677] LustreError: 69552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.510824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.114418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.580399] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1913.566894] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1920.747879] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.588080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1930.461505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.047340] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1946.603497] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.835196] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1969.638614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1969.668877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.692161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.685387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.686721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.698674] Lustre: Skipped 1 previous similar message [ 1971.731590] Lustre: Skipped 1 previous similar message [ 1974.513498] Lustre: server umount lustre-MDT0000 complete [ 1976.804649] LustreError: 71227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.839324] LustreError: 71227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1978.823285] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312302 with bad export cookie 11908494153906329575 [ 1978.836730] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.840769] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1979.179309] Lustre: server umount lustre-MDT0001 complete [ 1993.733757] Lustre: server umount lustre-OST0000 complete [ 2022.618256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.052469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2023.117684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2023.530748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.262685] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2037.064649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.188844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2037.250346] Lustre: 74425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.000869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.808955] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2053.246424] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.575762] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2068.278218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2068.654912] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2069.176654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.503504] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2085.613952] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2087.435634] Lustre: 76998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2089.346706] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2095.586342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.593720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2095.597128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.597136] Lustre: Skipped 1 previous similar message [ 2095.606747] LustreError: Skipped 1 previous similar message [ 2100.711928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.719807] Lustre: Skipped 1 previous similar message [ 2101.567475] Lustre: server umount lustre-OST0000 complete [ 2105.825708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.835814] Lustre: Skipped 2 previous similar messages [ 2105.836497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.844861] Lustre: Skipped 1 previous similar message [ 2111.937610] Lustre: server umount lustre-MDT0000 complete [ 2116.070393] LustreError: 75231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.081835] LustreError: 75231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2116.146789] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312440 with bad export cookie 11908494153906330422 [ 2116.170488] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.170642] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2116.692340] Lustre: server umount lustre-MDT0001 complete [ 2128.330612] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2132.377075] Key type lgssc unregistered [ 2132.956704] LNet: 78211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.963576] LNetError: 78211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.976415] LNet: Removed LNI 192.168.202.111@tcp [ 2134.539681] Key type .llcrypt unregistered [ 2134.543649] Key type ._llcrypt unregistered [ 2164.869816] Key type ._llcrypt registered [ 2164.871441] Key type .llcrypt registered [ 2164.952980] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2166.644316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.675716] alg: No test for adler32 (adler32-zlib) [ 2167.557225] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2167.953835] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2169.777379] Key type lgssc registered [ 2171.529735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.107073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.131956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.862234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.280443] LustreError: 79624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.298054] LustreError: 79624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2189.386168] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2193.380993] LustreError: 79625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.510058] LustreError: 79624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.117071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.541806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.837252] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2215.343740] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2222.108694] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2230.935478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.570665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.765057] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2248.701730] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2259.216959] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2260.087975] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2261.428864] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2264.402304] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2264.408420] Lustre: Skipped 2 previous similar messages [ 2268.623112] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2268.633920] Lustre: Skipped 3 previous similar messages [ 2277.549648] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2277.556945] Lustre: Skipped 7 previous similar messages [ 2293.615233] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2293.625716] Lustre: Skipped 13 previous similar messages [ 2326.471877] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2326.474576] Lustre: Skipped 31 previous similar messages [ 2376.894714] Lustre: 87746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2379.466488] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:58:22 (1786312702) [ 2381.027697] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2383.057719] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:58:25 (1786312705) [ 2388.448665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2388.456864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.476150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.520529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2391.532566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.538145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.552130] Lustre: Skipped 1 previous similar message [ 2396.648090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.661098] Lustre: Skipped 1 previous similar message [ 2401.774807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.781588] Lustre: Skipped 1 previous similar message [ 2402.271246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.773071] Lustre: server umount lustre-OST0000 complete [ 2409.440841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2409.448193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.464972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.003036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.332374] Lustre: server umount lustre-MDT0000 complete [ 2417.125852] LustreError: 80773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.135575] LustreError: 80773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2417.572896] LustreError: 84101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312741 with bad export cookie 650422708273914434 [ 2417.583553] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.585206] LustreError: 84101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2418.319992] Lustre: server umount lustre-MDT0001 complete [ 2430.498224] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2434.333361] Key type lgssc unregistered [ 2434.859545] LNet: 88999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.865959] LNetError: 88999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.877973] LNet: Removed LNI 192.168.202.111@tcp [ 2436.070074] Key type .llcrypt unregistered [ 2436.073328] Key type ._llcrypt unregistered [ 2443.698481] Key type ._llcrypt registered [ 2443.708173] Key type .llcrypt registered [ 2443.950879] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2449.727771] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules [ 2450.732358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2450.801621] alg: No test for adler32 (adler32-zlib) [ 2451.945827] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2452.242209] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2454.079894] Key type lgssc registered [ 2455.503425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2456.482367] LustreError: 90202:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2475.345616] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2478.180681] LustreError: 90202:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2485.552788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.580401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.412498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.161375] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2497.012141] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.033941] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2502.112888] LustreError: 91208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.869943] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:00:25 (1786312825) [ 2507.240815] LustreError: 91207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.041628] LustreError: 91239:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2508.066390] LustreError: 91239:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2508.074860] LustreError: 91239:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2508.589812] Lustre: server umount lustre-MDT0000 complete [ 2524.698171] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2527.754992] Key type lgssc unregistered [ 2528.034256] LNet: 92739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.038937] LNetError: 92739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.062209] LNet: Removed LNI 192.168.202.111@tcp [ 2530.280671] Key type .llcrypt unregistered [ 2530.286709] Key type ._llcrypt unregistered [ 2549.696891] Key type ._llcrypt registered [ 2549.700142] Key type .llcrypt registered [ 2549.885738] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2551.512025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.566918] alg: No test for adler32 (adler32-zlib) [ 2552.892948] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2553.291389] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2555.009969] Key type lgssc registered [ 2556.586434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2565.527851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.564829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.466658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.215350] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2577.895822] LustreError: 93972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.914044] LustreError: 93972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2582.453037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.645406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.863532] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2597.274731] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2604.006769] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2611.536350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.362217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.590995] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2627.676459] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2679.275666] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:03:22 (1786313002) [ 2681.515982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2683.396637] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:03:26 (1786313006) [ 2690.530404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.537703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.570015] Lustre: Skipped 2 previous similar messages [ 2690.589361] Lustre: Skipped 2 previous similar messages [ 2693.960537] Lustre: server umount lustre-MDT0000 complete [ 2695.649543] LustreError: 93973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.670105] LustreError: 93973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2698.422219] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313022 with bad export cookie 209347956685203863 [ 2698.432307] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.443015] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2698.992064] Lustre: server umount lustre-MDT0001 complete [ 2714.571481] Lustre: server umount lustre-OST0000 complete [ 2727.953266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.487564] LustreError: 98273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.651106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.309886] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2734.053805] LustreError: 98274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.146530] LustreError: 98273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.295455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.348080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.785142] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2756.736837] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2762.768801] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2776.494429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.744752] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2776.756594] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2776.794032] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2776.821464] LustreError: 100356:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2776.843151] LustreError: 100356:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2776.864969] LustreError: 100356:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2776.981835] Lustre: server umount lustre-OST0000 complete [ 2776.985131] LustreError: 100356:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2785.826481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.968361] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2785.974427] LustreError: 98261:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2785.999865] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2786.008945] LustreError: 100822:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2786.016715] LustreError: 100822:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2786.024571] LustreError: 100822:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2786.133637] Lustre: server umount lustre-OST0000 complete [ 2786.135610] LustreError: 100822:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2795.953426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.114352] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2796.123571] Lustre: Found index 0 for lustre-OST0000, updating log [ 2796.129580] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2796.220349] Lustre: lustre-OST0000: new disk, initializing [ 2796.226854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2796.237685] Lustre: 101359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2796.627889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.732583] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2807.634500] Lustre: Failing over lustre-OST0000 [ 2807.857884] Lustre: server umount lustre-OST0000 complete [ 2813.032958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.048453] Lustre: Skipped 1 previous similar message [ 2813.054073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.063169] Lustre: Skipped 1 previous similar message [ 2818.528255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.540229] Lustre: Skipped 1 previous similar message [ 2822.730679] Lustre: server umount lustre-MDT0000 complete [ 2823.660110] LustreError: 98268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.699180] LustreError: 98268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2827.364632] LustreError: 98255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313151 with bad export cookie 209347956685206607 [ 2827.373334] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.377247] LustreError: 98255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2827.849721] Lustre: server umount lustre-MDT0001 complete [ 2837.949205] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:06:00 (1786313160) [ 2855.825730] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2866.741094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.269270] LustreError: 103706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.455116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.091422] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2880.900704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.131549] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2889.430938] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2897.577938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.698739] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2905.595371] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2906.858460] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2906.869545] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2906.878664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2911.996034] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2912.007911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2912.025914] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2912.030235] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2912.044573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2914.297077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.383432] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2914.516854] Lustre: lustre-OST0001: new disk, initializing [ 2914.520620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2914.527942] Lustre: 106025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2914.824852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.828870] Lustre: Skipped 2 previous similar messages [ 2922.525648] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2923.102902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2923.115936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2923.296088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2930.124851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2938.568472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2943.970916] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.978647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2944.004091] Lustre: Skipped 2 previous similar messages [ 2944.014783] Lustre: Skipped 2 previous similar messages [ 2949.094509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2949.101701] Lustre: Skipped 1 previous similar message [ 2949.761257] Lustre: server umount lustre-MDT0001 complete [ 2950.955119] LustreError: 103703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.991652] LustreError: 103703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2960.589868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.696889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2960.702713] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2960.770938] Lustre: 103696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2962.610509] LustreError: 107572:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2962.621758] LustreError: 107572:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2962.649582] LustreError: 103698:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2962.751282] LustreError: 107584:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -17 [ 2962.763115] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2962.763115] [ 2967.041800] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2967.435843] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2978.682696] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:08:21 (1786313301) [ 2998.779485] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3022.736927] Lustre: 109641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3047.484418] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3051.690886] Lustre: 110776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3135.462669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.471932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.493661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.501319] Lustre: Skipped 1 previous similar message [ 3141.254265] Lustre: server umount lustre-MDT0000 complete [ 3145.154587] LustreError: 106024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313469 with bad export cookie 209347956685207601 [ 3145.164226] LustreError: 106024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3145.165221] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.398882] Lustre: server umount lustre-MDT0001 complete [ 3159.469205] Lustre: server umount lustre-OST0000 complete [ 3162.400996] Lustre: 93283:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786313470/real 1786313470] req@ffff9ef20d273800 x1873084811677056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786313486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3162.461691] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.474843] Lustre: Skipped 3 previous similar messages [ 3163.351829] Lustre: server umount lustre-OST0001 complete [ 3169.001055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3180.692566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3190.550627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3200.255736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3222.121893] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3233.305590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.315479] Lustre: lustre-MDT0000: reset Object Index mappings [ 3233.828313] LustreError: 114724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.835220] LustreError: 114724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3233.932689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.935174] Lustre: Skipped 1 previous similar message [ 3239.312275] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3249.152650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.187618] Lustre: lustre-MDT0001: reset Object Index mappings [ 3255.242386] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3257.997440] Lustre: 115875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3264.985068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.576810] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3274.113428] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3278.233937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3281.670083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.561315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3287.566667] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3289.391910] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3297.166070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3300.451462] Lustre: 117751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3355.808523] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:14:38 (1786313678) [ 3382.752385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3382.767382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.786739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.799719] Lustre: Skipped 7 previous similar messages [ 3386.750650] Lustre: server umount lustre-MDT0000 complete [ 3389.943819] LustreError: 114724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.966284] LustreError: 114724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3390.829568] LustreError: 114706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313714 with bad export cookie 209347956685254683 [ 3390.832358] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3390.846582] LustreError: 114706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3405.279146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3405.601748] Lustre: server umount lustre-MDT0001 complete [ 3415.962933] Lustre: server umount lustre-OST0000 complete [ 3426.771870] Lustre: server umount lustre-OST0001 complete [ 3435.287371] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 3446.578846] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3487.161784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.475894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3487.510853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3487.610291] Lustre: lustre-MDT0000: new disk, initializing [ 3487.731815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.738465] Lustre: Skipped 3 previous similar messages [ 3487.766490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3494.919326] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3505.109633] Lustre: server umount lustre-MDT0000 complete [ 3515.902345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.132745] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3516.149963] Lustre: 122022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3516.170930] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3521.661914] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3527.430084] Lustre: server umount lustre-MDT0000 complete [ 3538.465495] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:17:41 (1786313861) [ 3540.316669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3542.440344] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:17:45 (1786313865) [ 3562.375521] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3573.587132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.639207] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3589.432313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.598664] Lustre: 124073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3589.652580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3589.657384] Lustre: Skipped 1 previous similar message [ 3589.747558] Lustre: lustre-MDT0001: new disk, initializing [ 3589.899956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3589.914886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3595.246920] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3600.394553] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3608.323650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.749664] Lustre: lustre-OST0000: new disk, initializing [ 3608.753920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3608.771425] Lustre: 125650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3609.891737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.899399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3610.059233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3616.275103] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3630.114247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.367323] Lustre: lustre-OST0001: new disk, initializing [ 3630.372657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3630.383782] Lustre: 126521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3631.535580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3631.550958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3631.768463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3638.360734] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3649.769335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3652.893941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4049.181691] Lustre: 132948:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4049.189987] Lustre: 132948:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4049.197576] Lustre: 132948:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4049.208307] Lustre: 132948:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4049.217783] Lustre: 132948:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4049.224456] Lustre: 132948:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4311.863953] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:30:33 (1786314633) [ 4327.392279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4327.395088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.425769] Lustre: Skipped 5 previous similar messages [ 4327.437017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4327.453039] Lustre: Skipped 10 previous similar messages [ 4333.234413] Lustre: server umount lustre-MDT0000 complete [ 4334.055723] LustreError: 127328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.085718] LustreError: 127328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4337.437481] LustreError: 132961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314661 with bad export cookie 209347956685322765 [ 4337.439833] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.455334] LustreError: 132961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4339.169246] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.177426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4339.179353] Lustre: Skipped 4 previous similar messages [ 4339.185098] Lustre: Skipped 4 previous similar messages [ 4343.884426] Lustre: server umount lustre-MDT0001 complete [ 4363.232559] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4363.404128] Lustre: server umount lustre-OST0000 complete [ 4381.152505] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4381.361429] Lustre: server umount lustre-OST0001 complete [ 4388.163995] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 4399.561545] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4435.092071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.439552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4435.477792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4435.568919] Lustre: lustre-MDT0000: new disk, initializing [ 4435.723747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.736284] Lustre: Skipped 5 previous similar messages [ 4435.754048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4441.994974] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4452.775347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.861059] Lustre: 136000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4452.916675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4452.920994] Lustre: Skipped 1 previous similar message [ 4452.965759] Lustre: lustre-MDT0001: new disk, initializing [ 4453.170200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4453.191324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4458.219216] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4468.787221] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4474.377682] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4481.237983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4481.592458] Lustre: lustre-OST0000: new disk, initializing [ 4481.597654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4481.603782] Lustre: 137970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4481.749253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.762609] Lustre: Skipped 1 previous similar message [ 4483.233342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4483.253154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4483.415833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4488.433686] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4497.857209] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.849041] Lustre: lustre-MDD0000: changelog on [ 4510.770484] Lustre: lustre-MDD0001: changelog on [ 6301.197070] Lustre: lustre-MDD0001: changelog off [ 6303.243684] Lustre: 136005:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6304.150419] Lustre: lustre-MDD0000: changelog off [ 6311.392813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.400095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.403211] LustreError: Skipped 1 previous similar message [ 6311.415184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6311.429052] Lustre: Skipped 1 previous similar message [ 6311.434566] Lustre: Skipped 3 previous similar messages [ 6314.472405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.477180] Lustre: Skipped 1 previous similar message [ 6319.600228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.612242] Lustre: Skipped 1 previous similar message [ 6324.708604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.712236] Lustre: Skipped 1 previous similar message [ 6325.215410] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6325.403353] Lustre: server umount lustre-OST0000 complete [ 6329.826232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.854717] Lustre: Skipped 1 previous similar message [ 6334.948227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.956062] Lustre: Skipped 3 previous similar messages [ 6336.025360] Lustre: server umount lustre-MDT0000 complete [ 6340.067260] LustreError: 136011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.085459] LustreError: 136011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6340.348594] LustreError: 135992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316664 with bad export cookie 209347956686100899 [ 6340.352339] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.364558] LustreError: 135992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6340.744837] Lustre: server umount lustre-MDT0001 complete [ 6355.012133] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6358.130391] Key type lgssc unregistered [ 6358.447559] LNet: 147099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6358.465440] LNetError: 147099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6358.487488] LNet: Removed LNI 192.168.202.111@tcp [ 6359.484278] Key type .llcrypt unregistered [ 6359.487299] Key type ._llcrypt unregistered [ 6382.890655] Key type ._llcrypt registered [ 6382.893334] Key type .llcrypt registered [ 6383.002971] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 6403.098780] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6404.476214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.622972] alg: No test for adler32 (adler32-zlib) [ 6405.695648] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6406.061952] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6407.713533] Key type lgssc registered [ 6409.275934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6448.962663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6449.003521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.301583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6450.350817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6450.456330] Lustre: lustre-MDT0000: new disk, initializing [ 6450.532263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6450.549964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6456.140722] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6468.153757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.304931] Lustre: 150302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6468.404569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6468.409202] Lustre: Skipped 1 previous similar message [ 6468.470519] Lustre: lustre-MDT0001: new disk, initializing [ 6468.537246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6468.572650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6468.582921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6473.410684] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6483.054752] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6489.422764] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6496.419833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.681886] Lustre: lustre-OST0000: new disk, initializing [ 6496.688834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6496.697206] Lustre: 152269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6496.827572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.499653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6503.513316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6503.624591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6503.985411] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6515.123331] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6521.190629] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_module obdecho/obdecho [ 6523.290197] LustreError: 153276:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6523.299946] LustreError: 153276:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6525.285059] LustreError: 153372:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6539.750175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.752893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.761328] Lustre: Skipped 2 previous similar messages [ 6539.771333] Lustre: Skipped 2 previous similar messages [ 6543.168452] Lustre: server umount lustre-MDT0000 complete [ 6547.728208] LustreError: 150294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316871 with bad export cookie 5242791775872013248 [ 6547.735509] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6547.742463] LustreError: 150294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6548.077672] Lustre: server umount lustre-MDT0001 complete [ 6562.402269] Lustre: server umount lustre-OST0000 complete [ 6584.081609] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 6594.750892] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6631.500201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.779586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6631.817190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6631.929758] Lustre: lustre-MDT0000: new disk, initializing [ 6632.012466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.026067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6636.030813] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6646.218229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.317580] Lustre: 157092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6646.371172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6646.376406] Lustre: Skipped 1 previous similar message [ 6646.462286] Lustre: lustre-MDT0001: new disk, initializing [ 6646.566239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.627989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6646.647934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6651.393932] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6661.513991] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6667.129130] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6673.595198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6673.814610] Lustre: lustre-OST0000: new disk, initializing [ 6673.817668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6673.822178] Lustre: 159062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6673.911352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6679.590527] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6682.761464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6682.766414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.893426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.604639] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6694.857520] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6695.042668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6699.179386] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6699.302785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6703.587669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.593406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.595387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6703.618268] Lustre: Skipped 1 previous similar message [ 6708.715165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6708.722430] Lustre: Skipped 2 previous similar messages [ 6713.833255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6713.835982] Lustre: Skipped 1 previous similar message [ 6715.359170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.624731] Lustre: server umount lustre-OST0000 complete [ 6723.559463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6723.572887] LustreError: Skipped 1 previous similar message [ 6723.578608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.586624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6725.540768] Lustre: server umount lustre-MDT0000 complete [ 6728.975712] LustreError: 157084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317053 with bad export cookie 5242791775872014214 [ 6728.977245] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6728.981095] LustreError: 157084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6729.241380] Lustre: server umount lustre-MDT0001 complete [ 6733.417828] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:10:56 (1786317056) [ 6735.181339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6737.700637] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 19:11:00 (1786317060) [ 6756.605484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.121093] LustreError: 162035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.145892] LustreError: 162035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6757.200094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.382664] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6762.473416] LustreError: 162036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.587317] LustreError: 162035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6770.448404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.644810] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6783.902703] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6789.455612] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.309965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6796.732994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.744696] Lustre: Skipped 1 previous similar message [ 6803.664830] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6811.005848] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.535950] Lustre: *** cfs_fail_loc=718, val=0*** [ 6819.816632] LustreError: 164665:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000bf3e9d97) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6820.323952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.326792] LustreError: 162036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.334134] Lustre: Skipped 3 previous similar messages [ 6820.346449] LustreError: 162036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6821.911951] Lustre: server umount lustre-MDT0000 complete [ 6825.443050] LustreError: 162031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.482913] LustreError: 162031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6826.056426] LustreError: 163881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317150 with bad export cookie 5242791775872015096 [ 6826.067888] Lustre: *** cfs_fail_loc=718, val=0*** [ 6826.081810] Lustre: Skipped 3 previous similar messages [ 6826.091689] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.480376] Lustre: server umount lustre-MDT0001 complete [ 6835.564126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.703379] LustreError: 165364:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6836.710425] LustreError: 148524:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000975da81b x1873088851591168/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6836.967685] Lustre: *** cfs_fail_loc=723, val=0*** [ 6836.970938] Lustre: Skipped 1 previous similar message [ 6837.075785] LustreError: 165379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.098430] LustreError: 165393:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6837.114311] LustreError: 165393:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -122 [ 6837.124387] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6837.124387] [ 6837.136088] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6837.152952] LustreError: 165364:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6837.159516] LustreError: 165364:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -122 [ 6837.169291] LustreError: 165364:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6837.176128] LustreError: 165364:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6837.252693] Lustre: server umount lustre-MDT0000 complete [ 6837.256707] LustreError: 165364:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6845.212796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.358111] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.372736] LustreError: 162027:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6845.769663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.918325] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6846.444597] Lustre: 148526:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317154/real 1786317154] req@0000000054130e7a x1873088851590784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6847.519172] Lustre: 148526:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317155/real 1786317155] req@0000000060229243 x1873088851590912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6849.977717] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6850.866322] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6850.996295] Lustre: 148527:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317159/real 1786317159] req@00000000181d1533 x1873088851591040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6855.141071] LustreError: 165801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.165257] LustreError: 165801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6857.191150] Lustre: 148525:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317165/real 1786317165] req@00000000132b72c5 x1873088851592832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6857.228468] Lustre: 148525:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6860.231961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.534854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 6860.809495] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6862.094162] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6863.180033] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6865.905772] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6866.353164] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6866.463315] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6874.404282] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.091263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6882.172558] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6883.712658] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6887.824455] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 6901.729625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.733136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6901.748303] Lustre: Skipped 1 previous similar message [ 6901.764703] Lustre: Skipped 3 previous similar messages [ 6905.167513] Lustre: server umount lustre-OST0000 complete [ 6915.599356] Lustre: server umount lustre-MDT0000 complete [ 6917.089526] LustreError: 165801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.117294] LustreError: 165801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6919.433112] LustreError: 162015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317243 with bad export cookie 5242791775872016020 [ 6919.440496] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.444208] LustreError: 162015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6919.745389] Lustre: server umount lustre-MDT0001 complete [ 6931.688237] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 6934.938463] Key type lgssc unregistered [ 6935.205804] LNet: 168867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6935.215425] LNetError: 168867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6935.231703] LNet: Removed LNI 192.168.202.111@tcp [ 6936.180486] Key type .llcrypt unregistered [ 6936.182884] Key type ._llcrypt unregistered [ 6969.874103] Key type ._llcrypt registered [ 6969.875695] Key type .llcrypt registered [ 6970.058636] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 6971.862925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6972.043528] alg: No test for adler32 (adler32-zlib) [ 6973.198753] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6973.549622] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 6975.311197] Key type lgssc registered [ 6976.798756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6986.270567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6986.320402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.941794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.051310] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6998.508673] LustreError: 170737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6998.543146] LustreError: 170737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7003.114491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.634581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7008.519432] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7016.716465] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7022.169179] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7029.983951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7030.496399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.624665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7038.147110] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7048.443503] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.636992] Lustre: *** cfs_fail_loc=718, val=0*** [ 7056.686293] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 7057.986688] Lustre: server umount lustre-MDT0000 complete [ 7059.434394] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.439082] LustreError: 171466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.445568] Lustre: Skipped 1 previous similar message [ 7059.470506] LustreError: 171466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7061.702732] Lustre: *** cfs_fail_loc=718, val=0*** [ 7061.705893] LustreError: 170718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317385 with bad export cookie 785855333990530277 [ 7061.710836] Lustre: Skipped 3 previous similar messages [ 7061.753957] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.773271] LustreError: 170732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.821558] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 7062.096542] Lustre: server umount lustre-MDT0001 complete [ 7075.159412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.912860] Lustre: 170049:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317388/real 1786317388] req@00000000661292b9 x1873089446048000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7079.954527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.969703] Lustre: Skipped 1 previous similar message [ 7082.015528] Lustre: 170049:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317390/real 1786317390] req@00000000dae0f462 x1873089446048128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7085.024372] Lustre: 170047:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317393/real 1786317393] req@0000000041a20b80 x1873089446048256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7086.122148] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xae7eb433b5a0114 [ 7086.141660] Lustre: MGC192.168.202.111@tcp: Connection restored to 0@lo (at 0@lo) [ 7086.363380] Lustre: *** cfs_fail_loc=727, val=0*** [ 7086.417931] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (not set up) [ 7086.474719] LustreError: 174142:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-57) [ 7086.492509] LustreError: 174142:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -57 [ 7086.500748] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7086.500748] [ 7086.514449] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -57. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7086.528774] LustreError: 174112:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -57 [ 7086.537987] LustreError: 174112:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -57 [ 7086.551467] LustreError: 174112:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7086.560412] LustreError: 174112:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7086.684947] Lustre: server umount lustre-MDT0000 complete [ 7086.686958] LustreError: 174112:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -57 [ 7087.074791] Lustre: 170048:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317395/real 1786317395] req@00000000fb6dc6e4 x1873089446048512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7092.319099] Lustre: 170050:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786317400/real 1786317400] req@0000000014980e9f x1873089446048640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786317416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7096.700787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.779594] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7096.794814] LustreError: 170729:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7096.798923] LustreError: 174633:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.111@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000007c7b3168) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7097.054993] LustreError: 172568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.188388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.232406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7099.238396] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7101.774055] LustreError: 174646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.799546] LustreError: 174646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7102.219455] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7110.630193] LustreError: 174646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.648151] LustreError: 174646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7111.252291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.658818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7111.851388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7111.913833] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7111.956810] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7113.047191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7116.833383] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7117.289998] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7117.709940] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7117.803906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7125.527286] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7127.223429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7132.956745] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7134.381241] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7138.527190] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7151.031513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.049026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7151.054812] Lustre: Skipped 1 previous similar message [ 7155.789679] Lustre: server umount lustre-OST0000 complete [ 7161.318045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7161.323031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7161.331447] Lustre: Skipped 1 previous similar message [ 7161.341089] Lustre: Skipped 1 previous similar message [ 7165.757273] Lustre: server umount lustre-MDT0000 complete [ 7166.432665] LustreError: 174653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.487677] LustreError: 174653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7170.024095] LustreError: 170717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317494 with bad export cookie 785855333990531502 [ 7170.025892] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.044984] LustreError: 170717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7170.627808] Lustre: server umount lustre-MDT0001 complete [ 7185.828160] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7189.593732] Key type lgssc unregistered [ 7190.011255] LNet: 177723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7190.025092] LNetError: 177723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7190.049852] LNet: Removed LNI 192.168.202.111@tcp [ 7190.967576] Key type .llcrypt unregistered [ 7190.969587] Key type ._llcrypt unregistered [ 7215.080721] Key type ._llcrypt registered [ 7215.086163] Key type .llcrypt registered [ 7215.274806] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7217.047179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7217.151026] alg: No test for adler32 (adler32-zlib) [ 7218.651971] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7219.064659] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7220.879264] Key type lgssc registered [ 7222.502725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7231.795569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7231.817871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7233.550146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.695207] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7243.748726] LustreError: 179041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.769768] LustreError: 179041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7246.630828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.154382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7251.556341] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7259.006797] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7265.009253] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7271.782252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.297361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.455892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7278.551256] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7286.148294] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7293.923609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7293.949264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.965054] Lustre: Skipped 1 previous similar message [ 7298.347844] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 7298.530430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7298.540134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.554154] Lustre: Skipped 1 previous similar message [ 7299.346537] Lustre: server umount lustre-MDT0000 complete [ 7302.713793] LustreError: 179021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317626 with bad export cookie 14030006663746148114 [ 7302.716991] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7302.732786] LustreError: 179021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7303.445161] LustreError: 179037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.485747] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 7303.493750] Lustre: Skipped 4 previous similar messages [ 7304.171885] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.174457] LustreError: 179037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.175091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7307.199316] LustreError: 179037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7307.218684] Lustre: Skipped 1 previous similar message [ 7309.240837] Lustre: server umount lustre-MDT0001 complete [ 7316.186947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7329.228921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.475062] LustreError: 182882:0:(llog.c:1655:llog_backup()) MGC192.168.202.111@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7329.493788] Lustre: 182882:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.111@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7332.836771] LustreError: 182887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.864721] LustreError: 182887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7336.930403] LustreError: 182888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.968533] LustreError: 182888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7345.425473] LustreError: 182889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.450683] LustreError: 182889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7362.528425] LustreError: 182887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.546737] LustreError: 182887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7390.175844] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7390.185049] Lustre: 182882:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.111@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7395.603372] LustreError: 182888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.635965] LustreError: 182888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 7431.184532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7436.268560] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7436.606263] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7446.364908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.525229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7447.922776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.990738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 7453.176956] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7453.249595] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7461.065447] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7462.660980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7469.432752] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7471.470537] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7494.114895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7494.125862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.169184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.176255] Lustre: Skipped 3 previous similar messages [ 7495.522100] Lustre: server umount lustre-MDT0000 complete [ 7498.918073] LustreError: 181823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317823 with bad export cookie 14030006663746154785 [ 7498.929264] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7498.933468] LustreError: 181823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7499.267929] Lustre: server umount lustre-MDT0001 complete [ 7513.670288] Lustre: server umount lustre-OST0000 complete [ 7533.768504] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 7543.479921] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7589.156299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.551784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7589.608584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7589.722422] Lustre: lustre-MDT0000: new disk, initializing [ 7589.868844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.922161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7596.071482] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7609.601720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.970762] Lustre: 187986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7610.128118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7610.132263] Lustre: Skipped 1 previous similar message [ 7610.225960] Lustre: lustre-MDT0001: new disk, initializing [ 7610.383902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7610.513331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7610.526648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7615.449727] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7626.288544] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7632.431894] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.382090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.543348] Lustre: lustre-OST0000: new disk, initializing [ 7641.546700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7641.552617] Lustre: 189957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7649.053900] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7649.970176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7649.989335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7650.198247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7659.928782] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7666.019573] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7666.230755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7671.727514] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7671.947310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7675.885754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.893439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7675.899965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.900018] Lustre: Skipped 3 previous similar messages [ 7675.925562] Lustre: Skipped 1 previous similar message [ 7680.330541] Lustre: server umount lustre-OST0000 complete [ 7686.119986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.143957] Lustre: Skipped 1 previous similar message [ 7690.460352] Lustre: server umount lustre-MDT0000 complete [ 7691.244664] LustreError: 187993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7691.261579] LustreError: 187993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7694.150076] LustreError: 187977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318018 with bad export cookie 14030006663746171844 [ 7694.160134] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7694.167735] LustreError: 187977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7694.585206] Lustre: server umount lustre-MDT0001 complete [ 7704.220745] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:27:07 (1786318027) [ 7711.371494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.859113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.864960] Lustre: Skipped 1 previous similar message [ 7717.625460] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7728.708829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.132202] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7744.730473] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7750.872077] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7758.396806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.370198] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7773.984776] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7794.130764] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:28:37 (1786318117) [ 7800.801468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.806665] LustreError: Skipped 1 previous similar message [ 7800.811701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.822324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.825126] Lustre: Skipped 2 previous similar messages [ 7804.251211] Lustre: server umount lustre-OST0000 complete [ 7811.554622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7811.564801] LustreError: Skipped 1 previous similar message [ 7822.303164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.596326] Lustre: server umount lustre-MDT0000 complete [ 7826.526061] LustreError: 195369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318150 with bad export cookie 14030006663746172726 [ 7826.537564] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.538802] LustreError: 195369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7826.872487] Lustre: server umount lustre-MDT0001 complete [ 7839.822499] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7842.833479] Key type lgssc unregistered [ 7843.186205] LNet: 196124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7843.201935] LNetError: 196124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7843.220286] LNet: Removed LNI 192.168.202.111@tcp [ 7844.378451] Key type .llcrypt unregistered [ 7844.381557] Key type ._llcrypt unregistered [ 7866.966687] Key type ._llcrypt registered [ 7866.977988] Key type .llcrypt registered [ 7867.256036] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7869.420236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.618060] alg: No test for adler32 (adler32-zlib) [ 7871.172672] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7871.677782] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7873.535196] Key type lgssc registered [ 7875.858598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7886.889488] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7886.935661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.623510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7907.743344] LustreError: 197372:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 7912.863521] LustreError: 197372:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7912.891284] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7912.899544] LustreError: MGC192.168.202.111@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7912.921436] LustreError: 197372:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7912.938570] LustreError: 197372:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7912.946072] LustreError: 197372:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7913.031528] Lustre: server umount lustre-OST0000 complete [ 7913.034925] LustreError: 197372:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7922.997476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7924.469940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.584532] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7933.925161] LustreError: 197870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.948297] LustreError: 197870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7938.808370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.041561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7939.241620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7944.334203] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7952.141388] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.578349] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7965.741499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7966.029953] LustreError: 199688:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7966.054946] Lustre: 199688:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.111@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7966.283931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.164796] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7981.581481] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7984.736908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.760409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.766522] Lustre: Skipped 2 previous similar messages [ 7984.792893] Lustre: Skipped 2 previous similar messages [ 7990.062722] Lustre: server umount lustre-MDT0000 complete [ 7990.243245] LustreError: 197871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.274628] LustreError: 197871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7994.240484] LustreError: 197851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318318 with bad export cookie 3682153338083531402 [ 7994.250881] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.261229] LustreError: 197851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7994.417217] LustreError: 198600:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007dfd19e7 x1873090387668224/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7994.723981] Lustre: server umount lustre-MDT0001 complete [ 8009.472132] Lustre: server umount lustre-OST0000 complete [ 8020.638965] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 8033.202510] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8074.194533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8074.555323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8074.582101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8074.724266] Lustre: lustre-MDT0000: new disk, initializing [ 8074.874500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.890871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8080.363708] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8091.362780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.517588] Lustre: 203210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8091.595485] LustreError: 203223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.633917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8091.643358] Lustre: Skipped 1 previous similar message [ 8091.713460] Lustre: lustre-MDT0001: new disk, initializing [ 8091.796200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8091.826619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8091.843080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8096.011936] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8106.145520] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8112.959059] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8122.969958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8123.219698] Lustre: lustre-OST0000: new disk, initializing [ 8123.225197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8123.233661] Lustre: 205180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8123.317368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.622518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8129.638985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8129.775200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8131.301787] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8141.687837] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8146.813394] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8147.114982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8152.366087] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8152.691335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8155.623948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8155.624217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.636365] LustreError: Skipped 1 previous similar message [ 8155.638169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8155.679111] Lustre: Skipped 1 previous similar message [ 8160.743833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8160.754961] Lustre: Skipped 2 previous similar messages [ 8161.054279] Lustre: server umount lustre-OST0000 complete [ 8165.857642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.876716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8176.112066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8176.114442] Lustre: Skipped 4 previous similar messages [ 8180.195101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8180.578659] Lustre: server umount lustre-MDT0000 complete [ 8181.217957] LustreError: 203224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8181.236953] LustreError: 203224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8184.900856] LustreError: 204007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318509 with bad export cookie 3682153338083532123 [ 8184.901686] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8184.908138] LustreError: 204007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8185.389976] Lustre: server umount lustre-MDT0001 complete [ 8194.332924] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:35:17 (1786318517) [ 8217.874931] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8221.171516] Key type lgssc unregistered [ 8221.615083] LNet: 208138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8221.642686] LNetError: 208138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.703201] LNet: Removed LNI 192.168.202.111@tcp [ 8223.976431] Key type .llcrypt unregistered [ 8223.980120] Key type ._llcrypt unregistered [ 8249.284267] Key type ._llcrypt registered [ 8249.286467] Key type .llcrypt registered [ 8249.377981] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8251.178223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8251.246686] alg: No test for adler32 (adler32-zlib) [ 8252.394664] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8252.618105] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8254.458990] Key type lgssc registered [ 8255.585869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8264.319580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8264.343676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.028642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.603181] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8276.448689] LustreError: 209363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.467209] LustreError: 209363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8279.945480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.367247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.018760] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8295.109631] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8302.336544] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8311.475733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.672628] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8311.680792] LustreError: 211227:0:(llog.c:1696:llog_backup()) MGC192.168.202.111@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8311.687633] Lustre: 211227:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.111@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8311.877855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8320.273766] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8328.569634] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8331.744315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8331.753070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.775851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8335.330129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8335.335645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.344260] Lustre: Skipped 1 previous similar message [ 8335.368703] Lustre: Skipped 1 previous similar message [ 8337.202612] Lustre: server umount lustre-MDT0000 complete [ 8340.448790] LustreError: 209364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.490424] LustreError: 209364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8341.752985] LustreError: 209346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318665 with bad export cookie 2419765309188161242 [ 8341.762164] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8341.766373] LustreError: 209346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8342.437610] Lustre: server umount lustre-MDT0001 complete [ 8357.063651] Lustre: server umount lustre-OST0000 complete [ 8368.733753] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 8386.980931] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8434.796273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.294121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8435.328710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8435.448731] Lustre: lustre-MDT0000: new disk, initializing [ 8435.609317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8435.639835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8442.072923] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8454.482058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.666226] Lustre: 214774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8454.716501] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8454.719617] Lustre: Skipped 1 previous similar message [ 8454.722985] LustreError: 215557:0:(llog.c:1696:llog_backup()) MGC192.168.202.111@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8454.730274] LustreError: 215557:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8454.735335] Lustre: 215557:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.111@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8454.771416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8454.776675] Lustre: Skipped 1 previous similar message [ 8454.901148] Lustre: lustre-MDT0001: new disk, initializing [ 8455.060895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8455.141862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8455.155452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8461.432230] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8473.103687] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.075939] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8487.980170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.280541] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8488.283376] Lustre: Skipped 1 previous similar message [ 8488.286964] LustreError: 216719:0:(llog.c:1696:llog_backup()) MGC192.168.202.111@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8488.298213] LustreError: 216719:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8488.304511] Lustre: 216719:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.111@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8488.314392] Lustre: 216719:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8488.337128] Lustre: lustre-OST0000: new disk, initializing [ 8488.342961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8488.349917] Lustre: 216743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.524201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.211213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8494.220896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8494.369676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8495.360504] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8506.441917] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.766500] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8511.066432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8518.214574] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8518.450429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8525.284383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8525.290128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8525.302711] LustreError: Skipped 1 previous similar message [ 8525.303449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8525.303455] Lustre: Skipped 1 previous similar message [ 8525.360659] Lustre: Skipped 1 previous similar message [ 8526.895967] Lustre: server umount lustre-OST0000 complete [ 8531.952234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8531.966995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.979482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8531.981858] Lustre: Skipped 1 previous similar message [ 8540.643585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8540.647255] Lustre: Skipped 3 previous similar messages [ 8546.271548] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8546.642506] Lustre: server umount lustre-MDT0000 complete [ 8550.715397] LustreError: 215569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318874 with bad export cookie 2419765309188161956 [ 8550.720779] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8550.724029] LustreError: 215569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8551.108905] Lustre: server umount lustre-MDT0001 complete [ 8561.603413] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:41:24 (1786318884) [ 8587.872674] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8601.576274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.190986] LustreError: 219900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8602.280337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.711627] LustreError: 219901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.088658] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8612.840052] LustreError: 219900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8618.738889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.932180] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8628.419964] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8636.542831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8636.974533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.983870] Lustre: Skipped 1 previous similar message [ 8644.326922] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8653.789867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8654.033147] Lustre: lustre-OST0001: new disk, initializing [ 8654.043246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8654.057715] Lustre: 222208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8660.674253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8660.688431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8660.855861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8661.022856] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8673.869153] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8681.846129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8684.224633] Lustre: DEBUG MARKER: ADD OST3 [ 8690.176684] loop: module loaded [ 8690.338648] loop0: detected capacity change from 0 to 204800000 [ 8690.772571] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8690.780562] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8690.810445] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8698.504905] Lustre: DEBUG MARKER: STOP OST3 [ 8699.232971] loop0: detected capacity change from 0 to 204800000 [ 8707.250852] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.509533] Lustre: lustre-OST0003: new disk, initializing [ 8707.512577] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8707.516578] Lustre: 224294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8707.609457] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8707.614974] Lustre: Skipped 1 previous similar message [ 8709.430780] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8709.525264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8709.541478] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8709.597739] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8710.186382] Lustre: Failing over lustre-OST0003 [ 8710.528278] Lustre: server umount lustre-OST0003 complete [ 8711.456155] LustreError: 224422:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8714.619280] LustreError: 221387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.652343] LustreError: 221387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8714.731467] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8714.738656] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8714.773178] Lustre: Skipped 2 previous similar messages [ 8715.793758] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8727.665418] Lustre: DEBUG MARKER: START OST3 again [ 8734.992703] LustreError: 221385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.017313] LustreError: 221385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8735.362239] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8735.826551] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8737.578452] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8737.870060] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8737.876065] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8742.699195] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8766.649263] Lustre: Failing over lustre-OST0003 [ 8766.874553] Lustre: server umount lustre-OST0003 complete [ 8769.012836] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8769.014021] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.031546] LustreError: Skipped 1 previous similar message [ 8769.047015] LustreError: 221386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.047027] LustreError: 221386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8786.920248] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.925075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.955211] Lustre: Skipped 3 previous similar messages [ 8786.973255] Lustre: Skipped 4 previous similar messages [ 8793.182530] Lustre: server umount lustre-MDT0000 complete [ 8797.134517] LustreError: 224293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319121 with bad export cookie 2419765309188162838 [ 8797.150076] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8803.662551] Lustre: server umount lustre-MDT0001 complete [ 8807.769176] Lustre: server umount lustre-OST0000 complete [ 8812.020978] Lustre: server umount lustre-OST0001 complete [ 8819.297540] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 8828.673864] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8868.779893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.186712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8869.235381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8869.311165] Lustre: lustre-MDT0000: new disk, initializing [ 8869.422482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.425553] Lustre: Skipped 1 previous similar message [ 8869.448293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8874.657947] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8886.427117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8886.581447] Lustre: 229822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8886.671243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8886.675164] Lustre: Skipped 1 previous similar message [ 8886.841457] Lustre: lustre-MDT0001: new disk, initializing [ 8887.067773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8887.084904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8892.648926] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8903.803907] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8911.491736] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8919.572161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8919.878448] Lustre: lustre-OST0000: new disk, initializing [ 8919.881706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8919.888935] Lustre: 231792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8921.216648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8921.225742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8921.354224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8926.597858] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8937.526344] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8943.528378] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8943.779483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8949.384793] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8949.716413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8952.293047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8952.295924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.306561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.306570] Lustre: Skipped 8 previous similar messages [ 8952.338874] Lustre: Skipped 4 previous similar messages [ 8958.237182] Lustre: server umount lustre-OST0000 complete [ 8968.812392] Lustre: server umount lustre-MDT0000 complete [ 8972.781647] LustreError: 229828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8972.796591] LustreError: 229828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8973.091690] LustreError: 231803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319297 with bad export cookie 2419765309188165127 [ 8973.101173] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.106950] LustreError: 231803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8973.603898] Lustre: server umount lustre-MDT0001 complete [ 8977.571744] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:48:20 (1786319300) [ 8995.672972] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 9007.005852] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9048.121403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.442715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9048.473279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9048.554412] Lustre: lustre-MDT0000: new disk, initializing [ 9048.672331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9052.139252] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9062.506903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.646230] Lustre: 236402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9062.869778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9067.763927] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9077.523601] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9084.435611] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9092.300797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.787338] Lustre: 238373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9094.442726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9100.938956] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9111.225396] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9116.451849] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9116.752504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9121.926619] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9122.187839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9125.367186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.367611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9125.379331] Lustre: Skipped 2 previous similar messages [ 9125.391125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.392527] LustreError: Skipped 1 previous similar message [ 9125.406317] Lustre: Skipped 7 previous similar messages [ 9130.527965] Lustre: server umount lustre-OST0000 complete [ 9141.575928] Lustre: server umount lustre-MDT0000 complete [ 9145.749751] LustreError: 237200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319469 with bad export cookie 2419765309188166016 [ 9145.762529] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9145.763059] LustreError: 237200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9146.078741] Lustre: server umount lustre-MDT0001 complete [ 9156.931542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.468038] LustreError: 240589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.556354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.560566] Lustre: Skipped 5 previous similar messages [ 9163.218592] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9173.211064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.609781] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9187.196859] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9192.688078] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9199.618547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9206.373140] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9216.244840] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9227.744984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9227.754912] LustreError: Skipped 1 previous similar message [ 9227.768597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9227.773383] Lustre: Skipped 5 previous similar messages [ 9233.763126] Lustre: server umount lustre-OST0000 complete [ 9243.856496] Lustre: server umount lustre-MDT0000 complete [ 9247.878996] LustreError: 241326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319572 with bad export cookie 2419765309188166898 [ 9247.880869] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.888588] LustreError: 241326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9248.380420] Lustre: server umount lustre-MDT0001 complete [ 9356.644866] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9359.466872] Key type lgssc unregistered [ 9359.888513] LNet: 244262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9359.913231] LNetError: 244262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9359.972202] LNet: Removed LNI 192.168.202.111@tcp [ 9361.108556] Key type .llcrypt unregistered [ 9361.110709] Key type ._llcrypt unregistered [ 9390.675638] Key type ._llcrypt registered [ 9390.678859] Key type .llcrypt registered [ 9390.884072] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 9410.319607] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9411.410372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9411.435469] alg: No test for adler32 (adler32-zlib) [ 9412.475297] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 9412.705444] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9414.367208] Key type lgssc registered [ 9416.135967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9464.406084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9464.479910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.934214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9465.981259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9466.125768] Lustre: lustre-MDT0000: new disk, initializing [ 9466.230557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.276774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9472.853720] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9487.312370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.557201] Lustre: 247616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9487.639408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9487.643315] Lustre: Skipped 1 previous similar message [ 9487.759946] Lustre: lustre-MDT0001: new disk, initializing [ 9487.933359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9487.990143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9488.002198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9493.641910] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9503.680611] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9509.680569] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9518.973821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9519.351421] Lustre: lustre-OST0000: new disk, initializing [ 9519.361467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9519.374531] Lustre: 249588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9519.496723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.945510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9525.957275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9526.190139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9528.181956] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9539.194947] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9544.545469] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9544.901600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9550.486912] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9550.718277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9556.961034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9556.977702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.993292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9557.000274] Lustre: Skipped 1 previous similar message [ 9559.035677] Lustre: server umount lustre-OST0000 complete [ 9564.641814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9564.655543] LustreError: Skipped 1 previous similar message [ 9564.660657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.701736] Lustre: Skipped 1 previous similar message [ 9564.716400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.202162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.214077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.237051] Lustre: Skipped 1 previous similar message [ 9568.895754] Lustre: server umount lustre-MDT0000 complete [ 9572.446464] LustreError: 247607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319896 with bad export cookie 1682086007487483030 [ 9572.451095] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.463912] LustreError: 247607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9572.819762] Lustre: server umount lustre-MDT0001 complete [ 9603.185137] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9614.326262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.829063] LustreError: 252547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.850484] LustreError: 252547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9614.885082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.937058] LustreError: 252548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9620.436870] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9624.033357] LustreError: 252547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.157710] LustreError: 252548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9630.078699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.473537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9635.243671] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9638.413430] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9645.756054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9646.280261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9652.613421] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9661.749411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.953355] Lustre: lustre-OST0001: new disk, initializing [ 9661.956732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9661.962856] Lustre: 254856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9668.217403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9668.230866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9668.343189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9669.775501] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9682.286697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9692.306604] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9706.356191] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:00:29 (1786320029) [ 9709.543585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.550690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.570355] Lustre: Skipped 3 previous similar messages [ 9709.582396] Lustre: Skipped 3 previous similar messages [ 9714.596375] Lustre: server umount lustre-MDT0000 complete [ 9714.663843] LustreError: 253554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.690170] LustreError: 253554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9718.821909] LustreError: 254855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320042 with bad export cookie 1682086007487483905 [ 9718.833195] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.847351] LustreError: 254855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9719.431912] Lustre: server umount lustre-MDT0001 complete [ 9734.523063] Lustre: server umount lustre-OST0000 complete [ 9735.393255] Lustre: 245840:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786320043/real 1786320043] req@ffff9ef306dd7b80 x1873092003613440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786320059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9735.435923] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.739916] Lustre: server umount lustre-OST0001 complete [ 9746.699843] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 9758.918715] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9798.006536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.504732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9798.536164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9798.666986] Lustre: lustre-MDT0000: new disk, initializing [ 9798.783433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.794592] Lustre: Skipped 1 previous similar message [ 9798.827753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9804.326531] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9817.981531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.079418] Lustre: 258811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9818.132402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9818.135262] Lustre: Skipped 1 previous similar message [ 9818.191213] Lustre: lustre-MDT0001: new disk, initializing [ 9818.291075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9818.299689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9825.489211] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9835.960837] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9841.212819] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9847.706156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.979468] Lustre: lustre-OST0000: new disk, initializing [ 9847.983055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9847.988207] Lustre: 260780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9848.098358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9848.102291] Lustre: Skipped 1 previous similar message [ 9849.580194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9849.597289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9849.722600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9854.858812] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9864.906984] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9870.179706] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9870.560395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9875.824198] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9876.081332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9880.548544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.560279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.564225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.573715] Lustre: Skipped 1 previous similar message [ 9884.496741] Lustre: server umount lustre-OST0000 complete [ 9890.276664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9890.297224] LustreError: Skipped 1 previous similar message [ 9890.328963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9890.331701] Lustre: Skipped 1 previous similar message [ 9894.531699] Lustre: server umount lustre-MDT0000 complete [ 9895.914521] LustreError: 258818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.928234] LustreError: 258818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9897.959284] LustreError: 258805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320222 with bad export cookie 1682086007487486047 [ 9897.972849] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.983528] LustreError: 258805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9898.603337] Lustre: server umount lustre-MDT0001 complete [ 9906.900262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.570471] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9912.802114] LustreError: 262999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.841683] LustreError: 262999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9922.080227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9922.779292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9922.794050] Lustre: Skipped 1 previous similar message [ 9929.248730] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9938.479960] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9945.048523] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9952.397781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9952.742860] LustreError: 264821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.775587] LustreError: 264821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9958.342498] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9966.365293] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10116.076683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.084454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10116.107086] Lustre: Skipped 3 previous similar messages [10116.107497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.107502] Lustre: Skipped 1 previous similar message [10127.839834] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10128.048506] Lustre: server umount lustre-OST0000 complete [10132.448905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10139.179055] Lustre: server umount lustre-MDT0000 complete [10141.666264] LustreError: 262999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.678732] LustreError: 262999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10143.415710] LustreError: 262979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320467 with bad export cookie 1682086007487486929 [10143.422574] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10143.433336] LustreError: 262979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10143.848352] Lustre: server umount lustre-MDT0001 complete [10234.650266] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10238.153917] Key type lgssc unregistered [10238.521333] LNet: 266606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10238.526918] LNetError: 266606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10238.558298] LNet: Removed LNI 192.168.202.111@tcp [10239.558580] Key type .llcrypt unregistered [10239.562658] Key type ._llcrypt unregistered [10264.513372] Key type ._llcrypt registered [10264.516846] Key type .llcrypt registered [10264.617261] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [10282.651769] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10283.990315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10284.344675] alg: No test for adler32 (adler32-zlib) [10285.448884] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [10285.705321] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [10287.376693] Key type lgssc registered [10289.254258] Lustre: Echo OBD driver; http://www.lustre.org/ [10338.510911] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10349.249747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.290762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.591329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10350.616169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10350.706915] Lustre: lustre-MDT0000: new disk, initializing [10350.770463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.786712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10354.975357] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10365.460937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.703492] Lustre: 270683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10365.771993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10365.776457] Lustre: Skipped 1 previous similar message [10365.878107] Lustre: lustre-MDT0001: new disk, initializing [10366.016566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10366.050087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10366.079318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10371.488680] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10376.565405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10383.442917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.846232] Lustre: lustre-OST0000: new disk, initializing [10383.859194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10383.880661] Lustre: 272313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10384.053740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10389.089404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10389.106535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10389.289762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10391.211499] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10402.915926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.215910] Lustre: lustre-OST0001: new disk, initializing [10403.223469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10403.232664] Lustre: 273184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10403.387912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10410.106981] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10410.609736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10410.628828] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10410.746703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10422.193323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10432.696594] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10446.833971] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.848511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10446.849464] Lustre: Skipped 1 previous similar message [10446.867623] Lustre: Skipped 3 previous similar messages [10451.835313] Lustre: server umount lustre-MDT0000 complete [10455.668916] LustreError: 270675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320779 with bad export cookie 4326933895156256375 [10455.676479] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10455.677336] LustreError: 270675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10455.949756] Lustre: server umount lustre-MDT0001 complete [10470.014219] Lustre: server umount lustre-OST0000 complete [10473.441143] Lustre: 268191:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786320781/real 1786320781] req@ffff9ef20ca7b800 x1873092918983168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786320797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10473.500152] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10473.517170] Lustre: Skipped 2 previous similar messages [10474.630263] Lustre: server umount lustre-OST0001 complete [10491.483916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10502.857043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10513.696518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10525.310699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10550.109355] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10565.102447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.129159] Lustre: lustre-MDT0000: reset Object Index mappings [10565.551925] LustreError: 278200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10565.563530] LustreError: 278200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10565.650493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10570.723246] LustreError: 278201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.760537] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10575.845070] LustreError: 278200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.323586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.342338] Lustre: lustre-MDT0001: reset Object Index mappings [10580.903633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10586.043698] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10589.198513] Lustre: 279341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10596.315338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10603.662875] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10605.051969] LustreError: 279694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.168211] LustreError: 280117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.203986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10614.275330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10614.461618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10614.468290] Lustre: Skipped 1 previous similar message [10619.944033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10623.146891] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10631.154788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10635.326786] Lustre: 281210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10641.448588] Lustre: server umount lustre-MDT0000 complete [10642.401336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10642.422828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.436116] LustreError: 278201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10642.457230] LustreError: 278201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10645.476949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10645.517541] Lustre: Skipped 1 previous similar message [10645.785218] LustreError: 281212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320969 with bad export cookie 4326933895156263165 [10645.786184] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10645.802028] LustreError: 281212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10646.590820] Lustre: server umount lustre-MDT0001 complete [10661.463460] Lustre: server umount lustre-OST0000 complete [10675.947720] Lustre: server umount lustre-OST0001 complete [10696.319949] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:16:59 (1786321019) [10716.753919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [10727.830523] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10769.656771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.815783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10769.828143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10769.876514] Lustre: lustre-MDT0000: new disk, initializing [10769.938222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.959047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10775.707356] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10786.957660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.047858] Lustre: 285468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10787.093373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10787.096108] Lustre: Skipped 1 previous similar message [10787.151086] Lustre: lustre-MDT0001: new disk, initializing [10787.308935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10787.324227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10792.362175] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10801.280514] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10807.783082] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10815.349238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.739519] Lustre: lustre-OST0000: new disk, initializing [10815.744284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10815.762833] Lustre: 287440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10817.200741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10817.209578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10817.427752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10822.617127] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10832.786190] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10838.330351] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10838.564421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10843.045066] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10843.245146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10848.227687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.246033] Lustre: Skipped 1 previous similar message [10848.248925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10848.258913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10851.394480] Lustre: server umount lustre-OST0000 complete [10858.465438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.476756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10858.481520] Lustre: Skipped 2 previous similar messages [10858.490260] Lustre: Skipped 1 previous similar message [10861.386397] Lustre: server umount lustre-MDT0000 complete [10863.584264] LustreError: 287131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.606370] LustreError: 287131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10865.241936] LustreError: 286265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321189 with bad export cookie 4326933895156265258 [10865.246729] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10865.263398] LustreError: 286265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10865.792488] Lustre: server umount lustre-MDT0001 complete [10886.613336] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10898.530694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.948547] LustreError: 290374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.023981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10899.034681] Lustre: Skipped 2 previous similar messages [10904.280637] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10914.249404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.584329] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10922.533636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10930.085177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10937.211206] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10946.796955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10946.967755] Lustre: lustre-OST0001: new disk, initializing [10946.970940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10946.976217] Lustre: 292683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10948.397671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10948.416918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10948.534398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10953.545449] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10962.611376] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10965.351175] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10969.821651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10969.846430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10975.595608] Lustre: server umount lustre-MDT0000 complete [10976.225546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10976.240884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.256121] LustreError: 290375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.265863] LustreError: 290375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10979.787120] LustreError: 292693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321303 with bad export cookie 4326933895156266140 [10979.801420] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10979.804697] LustreError: 292693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10980.372590] Lustre: server umount lustre-MDT0001 complete [10995.215922] Lustre: server umount lustre-OST0000 complete [11009.719088] Lustre: server umount lustre-OST0001 complete [11029.178332] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11039.674626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.130780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11040.136880] Lustre: Skipped 3 previous similar messages [11044.642832] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11053.600622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.093150] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11062.238355] Lustre: 296878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11069.433900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11076.552729] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11085.329127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11085.718101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11091.519705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11093.351933] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11102.121547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11106.147604] Lustre: 298754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11111.289318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11111.371924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11121.763092] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:24:04 (1786321444) [11137.509329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11137.518327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11137.533981] Lustre: Skipped 5 previous similar messages [11137.564072] Lustre: Skipped 3 previous similar messages [11139.411219] Lustre: server umount lustre-MDT0000 complete [11142.631037] LustreError: 297531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11142.656825] LustreError: 297531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [11143.106389] LustreError: 295718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321467 with bad export cookie 4326933895156272440 [11143.109863] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11143.117144] LustreError: 295718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11147.748552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11149.580348] Lustre: server umount lustre-MDT0001 complete [11159.937343] Lustre: server umount lustre-OST0000 complete [11163.694319] Lustre: server umount lustre-OST0001 complete [11170.427705] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [11182.453708] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11240.610528] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11253.040738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.394351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11253.429600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11253.529117] Lustre: lustre-MDT0000: new disk, initializing [11253.648597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11259.051393] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11270.975527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.139295] Lustre: 303014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11271.670448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11277.955275] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11293.555098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.266918] Lustre: 304647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11295.727855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11302.909596] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11315.001313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11315.254729] Lustre: 305521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11315.399332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11315.401221] Lustre: Skipped 6 previous similar messages [11317.355626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11317.373348] Lustre: Skipped 1 previous similar message [11317.542366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11323.823975] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11333.814973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11337.802170] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11337.809698] Lustre: Skipped 1 previous similar message [11354.319637] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 20:27:57 (1786321677) [11358.693390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11358.693574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.704828] Lustre: Skipped 3 previous similar messages [11358.752222] Lustre: server umount lustre-MDT0000 complete [11362.268502] LustreError: 303007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321686 with bad export cookie 4326933895156280322 [11362.286691] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11362.290236] LustreError: 303007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11362.701791] Lustre: server umount lustre-MDT0001 complete [11376.883330] Lustre: server umount lustre-OST0000 complete [11379.686343] Lustre: 268190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786321687/real 1786321687] req@ffff9ef3412f6d80 x1873092919287680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786321703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11381.030154] Lustre: server umount lustre-OST0001 complete [11389.962018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.683838] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11400.189675] LustreError: 307767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.221027] LustreError: 307767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11405.442776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.479315] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11420.258368] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11427.425576] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11435.221371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11443.061381] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11451.031312] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11468.528859] loop1: detected capacity change from 0 to 204800000 [11468.743332] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11468.758160] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11468.784744] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11474.954631] loop1: detected capacity change from 0 to 204800000 [11480.901885] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11481.010545] LustreError: 307757:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1110: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11481.027131] LustreError: 307757:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1110 [11481.034770] LustreError: 307757:0:(mgs_handler.c:517:mgs_target_reg()) fs1110-MDT0001: Error checking target index: rc = -13 [11481.044550] LustreError: fs1110-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11481.051337] LustreError: 311488:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11481.055362] LustreError: 311488:0:(tgt_mount.c:2129:server_put_super()) no obd fs1110-MDT0001 [11481.059212] LustreError: 311488:0:(tgt_mount.c:116:server_deregister_mount()) fs1110-MDT0001 not registered [11481.093344] Lustre: server umount fs1110-MDT0001 complete [11481.095592] LustreError: 311488:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11490.466592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.599485] Lustre: Setting parameter fs1110-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1110-MDT0001 [11490.660357] Lustre: srv-fs1110-MDT0001: No data found on store. Initialize space. [11490.667244] Lustre: Skipped 4 previous similar messages [11490.803153] Lustre: fs1110-MDT0001: new disk, initializing [11490.805960] Lustre: Skipped 3 previous similar messages [11490.948640] Lustre: srv-fs1110-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11490.964908] Lustre: 311955:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1110-MDT0001-mdtlov: can't create new llog: rc = -115 [11490.985035] LustreError: 311955:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11495.526764] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11502.367227] Lustre: server umount fs1110-MDT0001 complete [11516.581816] loop1: detected capacity change from 0 to 204800000 [11516.918027] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11522.416633] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11522.992735] Lustre: 313520:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1110-MDT0001-mdtlov: can't create new llog: rc = -115 [11523.006339] LustreError: 313520:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11527.534531] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11532.269386] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11532.375580] Lustre: server umount fs1110-MDT0001 complete [11543.872569] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [11543.880856] Lustre: Skipped 3 previous similar messages [11544.033140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.048368] Lustre: Skipped 4 previous similar messages [11557.347962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11557.925732] Lustre: server umount lustre-MDT0000 complete [11568.024425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.300584] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11574.295781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11575.533209] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11586.569733] loop1: detected capacity change from 0 to 204800000 [11593.979625] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.250809] Lustre: 315969:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1110-MDT0001-mdtlov: can't create new llog: rc = -115 [11594.258147] LustreError: 315969:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11600.032765] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11604.993941] Lustre: server umount fs1110-MDT0001 complete [11629.537739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11629.545987] LustreError: Skipped 1 previous similar message [11629.556962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11629.563496] Lustre: Skipped 9 previous similar messages [11633.890312] Lustre: server umount lustre-OST0000 complete [11644.580364] Lustre: server umount lustre-MDT0000 complete [11648.732063] LustreError: 317765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321972 with bad export cookie 4326933895156283003 [11648.742115] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11648.744328] LustreError: 317765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11649.224090] Lustre: server umount lustre-MDT0001 complete [11660.660389] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [11663.655651] Key type lgssc unregistered [11663.998283] LNet: 318318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11664.011857] LNetError: 318318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11664.026727] LNet: Removed LNI 192.168.202.111@tcp [11665.034293] Key type .llcrypt unregistered [11665.036061] Key type ._llcrypt unregistered [11688.302638] Key type ._llcrypt registered [11688.304397] Key type .llcrypt registered [11688.383609] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [11689.642489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11689.829641] alg: No test for adler32 (adler32-zlib) [11691.346262] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11691.816843] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [11693.584365] Key type lgssc registered [11695.919362] Lustre: Echo OBD driver; http://www.lustre.org/ [11705.994444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11706.018377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.630629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11713.692122] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11717.092965] LustreError: 319593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.127457] LustreError: 319593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11722.209538] LustreError: 319592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.752631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.372287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11728.604119] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11738.212654] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11745.969073] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11754.491834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11755.062389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11762.159835] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11769.391301] LustreError: 321425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.305665] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.460238] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11778.605778] LustreError: 321425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11778.629251] LustreError: 321425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11787.538417] LustreError: 321425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.559061] LustreError: 321425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11800.676822] loop1: detected capacity change from 0 to 204800000 [11800.927244] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11800.934450] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11800.952893] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11804.145779] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.187526] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11806.446438] loop1: detected capacity change from 0 to 204800000 [11812.434668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11812.564477] Lustre: fs27546-OST0000: new disk, initializing [11812.568830] Lustre: srv-fs27546-OST0000: No data found on store. Initialize space. [11812.576729] Lustre: 323353:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11812.677652] Lustre: fs27546-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11819.266111] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11827.372332] Lustre: server umount fs27546-OST0000 complete [11838.758353] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11838.810681] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11843.687959] loop1: detected capacity change from 0 to 204800000 [11845.106925] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11850.165479] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11850.670293] Lustre: fs27546-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.381369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11858.064632] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11867.201376] loop2: detected capacity change from 0 to 204800000 [11867.453780] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11867.478037] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11867.497707] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11873.808304] loop2: detected capacity change from 0 to 204800000 [11879.772422] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.932297] LustreError: fs27546-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11879.946504] LustreError: fs27546-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11879.952126] LustreError: 326394:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11879.959165] LustreError: 326394:0:(tgt_mount.c:2129:server_put_super()) no obd fs27546-OST0001 [11879.966366] LustreError: 326394:0:(tgt_mount.c:116:server_deregister_mount()) fs27546-OST0001 not registered [11880.006390] Lustre: server umount fs27546-OST0001 complete [11880.010084] LustreError: 326394:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11892.008751] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11892.282962] Lustre: fs27546-OST0001: new disk, initializing [11892.294518] Lustre: srv-fs27546-OST0001: No data found on store. Initialize space. [11892.312358] Lustre: 326946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11892.402922] Lustre: fs27546-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11899.559991] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11905.334480] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.358144] LustreError: 321427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11908.721656] Lustre: server umount fs27546-OST0000 complete [11919.509648] Lustre: server umount fs27546-OST0001 complete [11933.779216] loop1: detected capacity change from 0 to 204800000 [11940.126596] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11940.453770] Lustre: fs27546-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11947.590515] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11954.692677] loop2: detected capacity change from 0 to 204800000 [11960.959234] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.024889] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [11979.355528] Lustre: server umount fs27546-OST0000 complete [11991.822877] Lustre: server umount fs27546-OST0001 complete [11998.702814] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12007.907202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12007.915108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.939502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12008.936333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12008.937615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.715654] Lustre: server umount lustre-OST0000 complete [12016.096342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12016.107203] LustreError: Skipped 1 previous similar message [12016.110606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12016.116736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12016.121065] Lustre: Skipped 1 previous similar message [12019.174916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.183767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.330765] Lustre: server umount lustre-MDT0000 complete [12025.986466] LustreError: 319574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322350 with bad export cookie 7597633481535520925 [12025.996308] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12025.999551] LustreError: 319574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12026.486728] Lustre: server umount lustre-MDT0001 complete [12037.689412] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [12040.270383] Key type lgssc unregistered [12040.542576] LNet: 332875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12040.549556] LNetError: 332875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12040.572672] LNet: Removed LNI 192.168.202.111@tcp [12041.517204] Key type .llcrypt unregistered [12041.520780] Key type ._llcrypt unregistered [12071.901289] Key type ._llcrypt registered [12071.906952] Key type .llcrypt registered [12072.073558] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [12088.198774] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [12089.750461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12089.831646] alg: No test for adler32 (adler32-zlib) [12090.870869] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12091.061161] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [12092.783187] Key type lgssc registered [12093.923126] Lustre: Echo OBD driver; http://www.lustre.org/ [12130.694928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12130.737477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12132.033891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12132.082861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12132.199466] Lustre: lustre-MDT0000: new disk, initializing [12132.308071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12132.322937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12137.555580] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12148.113716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.295244] Lustre: 336736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12148.360731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12148.363639] Lustre: Skipped 1 previous similar message [12148.438647] Lustre: lustre-MDT0001: new disk, initializing [12148.553426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.609108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12148.632265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12153.601035] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12163.464652] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12169.110958] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12175.382422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12175.619494] Lustre: lustre-OST0000: new disk, initializing [12175.623579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12175.631504] Lustre: 338705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12175.741325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12182.374510] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12185.700739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12185.708603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12185.868238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12192.675963] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12194.842523] Lustre: Mounted lustre-client - version 2.17.56_55_g704561f [12195.780498] Lustre: Unmounted lustre-client [12206.561758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12206.561936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.576407] LustreError: Skipped 1 previous similar message [12206.582155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.600231] Lustre: Skipped 1 previous similar message [12206.620956] Lustre: Skipped 1 previous similar message [12210.148266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12210.158205] Lustre: Skipped 1 previous similar message [12212.446071] Lustre: server umount lustre-OST0000 complete [12220.385106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12220.407246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.417225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12220.434751] Lustre: Skipped 1 previous similar message [12223.520961] Lustre: server umount lustre-MDT0000 complete [12227.230157] LustreError: 336729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322551 with bad export cookie 1089799286718692953 [12227.247284] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12227.251960] LustreError: 336729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12227.591131] Lustre: server umount lustre-MDT0001 complete [12239.743855] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [12242.290833] Key type lgssc unregistered [12247.712494] LNet: 340871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12247.728275] LNetError: 340871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12326.570411] LNet: Removed LNI 192.168.202.111@tcp [12327.692766] Key type .llcrypt unregistered [12327.702441] Key type ._llcrypt unregistered [12346.361723] Key type ._llcrypt registered [12346.367771] Key type .llcrypt registered [12346.521390] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [12363.832443] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [12365.274379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12365.474469] alg: No test for adler32 (adler32-zlib) [12366.495525] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12366.806994] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [12368.479185] Key type lgssc registered [12369.760581] Lustre: Echo OBD driver; http://www.lustre.org/ [12402.201478] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:45:25 (1786322725) [12408.975059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12409.015136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.313611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12410.347388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12410.433889] Lustre: lustre-MDT0000: new disk, initializing [12410.536522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12410.572200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12415.233585] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12426.970212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12427.066967] Lustre: 344094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12427.111880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12427.115774] Lustre: Skipped 1 previous similar message [12427.172426] Lustre: lustre-MDT0001: new disk, initializing [12427.272422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12427.309587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12427.315565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12432.214407] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12444.187387] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12450.759371] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12458.367271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12458.624780] Lustre: lustre-OST0000: new disk, initializing [12458.629915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12458.635087] Lustre: 346064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12458.713776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.235242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12465.251950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12465.387565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12465.638148] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12476.106601] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12496.355445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.356717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12496.357255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12496.367808] Lustre: Skipped 1 previous similar message [12496.372126] LustreError: Skipped 1 previous similar message [12499.524874] Lustre: server umount lustre-OST0000 complete [12504.035684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12504.051241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.081063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.090094] Lustre: Skipped 1 previous similar message [12506.592327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12506.594095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12511.712816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12511.717861] Lustre: Skipped 2 previous similar messages [12516.839030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12516.847133] Lustre: Skipped 1 previous similar message [12518.367444] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12518.592306] Lustre: server umount lustre-MDT0000 complete [12522.118743] LustreError: 344087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322846 with bad export cookie 15615552180874439136 [12522.119033] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12522.125043] LustreError: 344087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12522.479914] Lustre: server umount lustre-MDT0001 complete [12532.190758] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [12535.059562] Key type lgssc unregistered [12535.404650] LNet: 347977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12535.410749] LNetError: 347977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12536.487352] LNet: Removed LNI 192.168.202.111@tcp [12537.297326] Key type .llcrypt unregistered [12537.299487] Key type ._llcrypt unregistered [12557.591987] Key type ._llcrypt registered [12557.596177] Key type .llcrypt registered [12557.818051] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [12559.219037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12559.353171] alg: No test for adler32 (adler32-zlib) [12560.649718] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12560.931849] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [12562.623253] Key type lgssc registered [12563.778445] Lustre: Echo OBD driver; http://www.lustre.org/ [12571.228803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12571.253721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12572.626278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12577.455995] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12582.883134] LustreError: 349251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12582.892828] LustreError: 349251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12585.485539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.159429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12590.466880] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12599.059832] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12605.705675] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12613.313138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12613.606021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12620.048188] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12627.131736] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12639.115740] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12640.557815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12644.139083] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [12646.379050] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12646.386213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12646.395199] Lustre: Skipped 1 previous similar message [12646.416031] Lustre: Skipped 2 previous similar messages [12649.228742] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [12650.330065] Lustre: server umount lustre-MDT0000 complete [12651.494461] LustreError: 349979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.541423] LustreError: 349979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12654.057330] LustreError: 349232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322978 with bad export cookie 6772987785884004831 [12654.062725] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12654.066491] LustreError: 349232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12654.350114] LustreError: 349248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.373919] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [12656.608063] LustreError: 349986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12656.609429] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12656.647699] Lustre: Skipped 1 previous similar message [12659.467800] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [12659.473235] Lustre: Skipped 1 previous similar message [12664.596827] LustreError: 349246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.613028] LustreError: 349246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12668.385285] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12668.566627] Lustre: server umount lustre-MDT0001 complete [12673.481625] Lustre: server umount lustre-OST0000 complete [12683.084365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.634262] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12693.288394] Lustre: 353433:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.111@tcp [12695.296639] Lustre: server umount MGS complete [12703.812623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12704.300541] LustreError: 354005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.316683] LustreError: 354005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12704.358196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12709.074971] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12719.214849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12719.552225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12724.418957] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12732.599962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.054145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12738.841663] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12746.835276] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12748.568291] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12755.287358] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12756.858625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12763.116132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12763.125759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12763.143523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12763.152613] Lustre: Skipped 3 previous similar messages [12765.153248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12765.179808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.327949] Lustre: server umount lustre-OST0000 complete [12775.393160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.407783] Lustre: Skipped 1 previous similar message [12777.731469] Lustre: server umount lustre-MDT0000 complete [12780.518756] LustreError: 355008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12780.532081] LustreError: 355008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12781.689137] LustreError: 353985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323105 with bad export cookie 6772987785884005671 [12781.693074] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12781.696754] LustreError: 353985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12782.201578] Lustre: server umount lustre-MDT0001 complete [12795.893679] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [12799.070850] Key type lgssc unregistered [12799.393816] LNet: 357449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12799.399823] LNetError: 357449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12799.420500] LNet: Removed LNI 192.168.202.111@tcp [12800.498292] Key type .llcrypt unregistered [12800.499751] Key type ._llcrypt unregistered [12819.737274] Key type ._llcrypt registered [12819.741809] Key type .llcrypt registered [12819.887151] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [12837.969197] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [12839.311916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12839.366701] alg: No test for adler32 (adler32-zlib) [12840.508924] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12840.809053] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [12842.471175] Key type lgssc registered [12844.284963] Lustre: Echo OBD driver; http://www.lustre.org/ [12895.442561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12895.539592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12896.946409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12896.999657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12897.126581] Lustre: lustre-MDT0000: new disk, initializing [12897.236731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12897.268378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12902.897268] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12915.399639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12915.533319] Lustre: 360615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12915.600477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12915.608162] Lustre: Skipped 1 previous similar message [12915.716871] Lustre: lustre-MDT0001: new disk, initializing [12915.818452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12915.865472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12915.878899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12921.370779] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12931.360175] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12937.029334] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12945.294430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12945.769828] Lustre: lustre-OST0000: new disk, initializing [12945.778198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12945.783710] Lustre: 362582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12945.933788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12952.764920] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [12953.210488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12953.225477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12953.393479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12963.928375] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12969.589671] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12969.830800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12976.098609] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12976.286759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12979.171851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12979.172313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12979.201550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12979.215126] Lustre: Skipped 1 previous similar message [12979.226724] Lustre: Skipped 1 previous similar message [12984.291039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12984.300076] Lustre: Skipped 1 previous similar message [12984.789874] Lustre: server umount lustre-OST0000 complete [12992.482802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12992.490151] LustreError: Skipped 1 previous similar message [12992.493871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12992.509545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12994.528508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12994.529237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12996.018731] Lustre: server umount lustre-MDT0000 complete [13000.458436] LustreError: 360607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323324 with bad export cookie 11262041295647668905 [13000.463519] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13000.472726] LustreError: 360607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13000.912713] Lustre: server umount lustre-MDT0001 complete [13011.867286] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:55:34 (1786323334) [13030.037381] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [13041.369181] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [13076.590301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13076.950467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13076.994089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13077.100739] Lustre: lustre-MDT0000: new disk, initializing [13077.165863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13077.194616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13082.271973] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [13092.724594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13092.840064] Lustre: 367344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13092.891369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13092.894912] Lustre: Skipped 1 previous similar message [13092.953518] Lustre: lustre-MDT0001: new disk, initializing [13093.021112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13093.052639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13093.069820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13097.782974] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [13107.031578] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13112.915811] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13119.884441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13120.173585] Lustre: lustre-OST0000: new disk, initializing [13120.183664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13120.190847] Lustre: 369313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13120.299431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13126.196096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13126.206414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13126.350075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13126.853225] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [13137.338478] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13177.824519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13177.833378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.840332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13177.849994] Lustre: Skipped 1 previous similar message [13177.863160] Lustre: Skipped 2 previous similar messages [13188.065765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13188.075074] Lustre: Skipped 2 previous similar messages [13190.111990] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13190.532774] Lustre: server umount lustre-OST0000 complete [13195.746710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13195.759170] LustreError: Skipped 1 previous similar message [13195.768734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13200.279859] Lustre: server umount lustre-MDT0000 complete [13203.425610] LustreError: 367349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13203.450427] LustreError: 367349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13204.276890] LustreError: 369312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323528 with bad export cookie 11262041295647669787 [13204.277815] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13204.293558] LustreError: 369312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13204.866967] Lustre: server umount lustre-MDT0001 complete [13214.739427] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13217.997512] Key type lgssc unregistered [13218.293140] LNet: 371494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13218.306769] LNetError: 371494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13218.326148] LNet: Removed LNI 192.168.202.111@tcp [13219.345426] Key type .llcrypt unregistered [13219.347192] Key type ._llcrypt unregistered [13320.378570] Key type ._llcrypt registered [13320.382655] Key type .llcrypt registered [13320.577958] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13321.153694] Key type .llcrypt unregistered [13321.163667] Key type ._llcrypt unregistered [13408.890529] Key type ._llcrypt registered [13408.891871] Key type .llcrypt registered [13409.091539] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13409.569967] Key type .llcrypt unregistered [13409.571655] Key type ._llcrypt unregistered [13446.385885] Key type ._llcrypt registered [13446.387622] Key type .llcrypt registered [13446.484064] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13446.862461] Key type .llcrypt unregistered [13446.867540] Key type ._llcrypt unregistered [13477.774293] Key type ._llcrypt registered [13477.778223] Key type .llcrypt registered [13477.915899] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13478.362889] Key type .llcrypt unregistered [13478.370889] Key type ._llcrypt unregistered [13564.917680] Key type ._llcrypt registered [13564.919185] Key type .llcrypt registered [13565.008361] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13565.310515] Key type .llcrypt unregistered [13565.317303] Key type ._llcrypt unregistered [13597.287688] Key type ._llcrypt registered [13597.290869] Key type .llcrypt registered [13597.404668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13597.763555] Key type .llcrypt unregistered [13597.767537] Key type ._llcrypt unregistered [13628.751605] Key type ._llcrypt registered [13628.757362] Key type .llcrypt registered [13628.865262] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13629.324411] Key type .llcrypt unregistered [13629.326797] Key type ._llcrypt unregistered [13663.287807] Key type ._llcrypt registered [13663.293222] Key type .llcrypt registered [13663.418221] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13663.825362] Key type .llcrypt unregistered [13663.826867] Key type ._llcrypt unregistered [13697.612429] Key type ._llcrypt registered [13697.614157] Key type .llcrypt registered [13697.735878] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13698.067596] Key type .llcrypt unregistered [13698.070981] Key type ._llcrypt unregistered [13728.434452] Key type ._llcrypt registered [13728.439440] Key type .llcrypt registered [13728.532167] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13728.819439] Key type .llcrypt unregistered [13728.821550] Key type ._llcrypt unregistered [13794.258671] Key type ._llcrypt registered [13794.260948] Key type .llcrypt registered [13794.397838] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13794.715416] Key type .llcrypt unregistered [13794.716956] Key type ._llcrypt unregistered [13827.139362] Key type ._llcrypt registered [13827.143071] Key type .llcrypt registered [13827.239830] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13827.579448] Key type .llcrypt unregistered [13827.581288] Key type ._llcrypt unregistered [13858.072529] Key type ._llcrypt registered [13858.076541] Key type .llcrypt registered [13858.225376] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [13858.601410] Key type .llcrypt unregistered [13858.602982] Key type ._llcrypt unregistered [13906.502296] Key type ._llcrypt registered [13906.504367] Key type .llcrypt registered [13906.593875] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [13922.387666] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [13923.771374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13923.854172] alg: No test for adler32 (adler32-zlib) [13925.034651] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [13925.362232] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [13927.152276] Key type lgssc registered [13928.416851] Lustre: Echo OBD driver; http://www.lustre.org/ [13962.227624] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13962.265408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13963.510669] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13963.547084] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13963.638491] Lustre: tst250fs-MDT0000: new disk, initializing [13963.776502] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13963.808516] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13968.462586] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [13978.704024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13978.860329] Lustre: 391890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13978.913929] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13978.918317] Lustre: Skipped 1 previous similar message [13979.014682] Lustre: tst250fs-MDT0001: new disk, initializing [13979.101280] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13979.136954] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13979.153124] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13983.633124] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [13987.942792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13993.294665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13993.605754] Lustre: tst250fs-OST0000: new disk, initializing [13993.615419] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13993.621363] Lustre: 393522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13993.720886] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13999.163514] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13999.174564] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13999.319973] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13999.898928] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [14010.797069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14010.964620] Lustre: tst250fs-OST0001: new disk, initializing [14010.968782] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14010.976191] Lustre: 394392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14011.027063] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14017.653817] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [14019.260512] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14019.277137] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14019.368411] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14023.049865] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14025.531403] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14025.534473] Lustre: Skipped 2 previous similar messages [14045.154583] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14045.172879] Lustre: Skipped 1 previous similar message [14045.173154] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14045.194884] Lustre: Skipped 3 previous similar messages [14047.677262] Lustre: server umount tst250fs-MDT0000 complete [14050.887945] LustreError: 392684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786324375 with bad export cookie 18365964669714886251 [14050.901084] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14050.903436] LustreError: 392684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14051.290728] Lustre: server umount tst250fs-MDT0001 complete [14064.800564] Lustre: server umount tst250fs-OST0000 complete [14079.006651] Lustre: server umount tst250fs-OST0001 complete [14096.081959] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [14103.914637] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [14133.516536] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 21:14:16 (1786324456) [14135.127071] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14150.208507] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [14158.940177] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [14191.273337] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13954 sec ========= 21:15:14 (1786324514) [14192.970381] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:15:16 (1786324516) === [14196.164487] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:15:19 (1786324519) === [14237.705436] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [14240.266647] Key type lgssc unregistered [14240.611751] LNet: 403537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14240.628380] LNetError: 403537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14240.650277] LNet: Removed LNI 192.168.202.111@tcp [14241.485773] Key type .llcrypt unregistered [14241.488047] Key type ._llcrypt unregistered