[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533764549 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003314] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008565] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011162] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.023200] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032125] *** VALIDATE tmpfs *** [ 0.034204] *** VALIDATE proc *** [ 0.035269] *** VALIDATE cgroup *** [ 0.036015] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044303] debug: unmapping init [mem 0xffffffffab859000-0xffffffffab860fff] [ 0.046198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050022] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054018] ... event mask: 000000070000000f [ 0.055284] rcu: Hierarchical SRCU implementation. [ 0.057498] smp: Bringing up secondary CPUs ... [ 0.058693] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062324] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141299] node 0 deferred pages initialised in 74ms [ 0.144639] devtmpfs: initialized [ 0.146084] x86/mm: Memory block size: 128MB [ 0.148785] gcov: version magic: 0x41383552 [ 0.150333] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154323] pinctrl core: initialized pinctrl subsystem [ 0.155211] [ 0.155903] ************************************************************* [ 0.156016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157014] ** ** [ 0.158015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159015] ** ** [ 0.160015] ** This means that this kernel is built to expose internal ** [ 0.161014] ** IOMMU data structures, which may compromise security on ** [ 0.162015] ** your system. ** [ 0.163015] ** ** [ 0.164014] ** If you see this message and you are not debugging the ** [ 0.165014] ** kernel, report this immediately to your vendor! ** [ 0.166015] ** ** [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168017] ************************************************************* [ 0.169718] NET: Registered protocol family 16 [ 0.170450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173528] cpuidle: using governor menu [ 0.174867] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177559] PCI: Using configuration type 1 for base access [ 0.179123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190066] cryptd: max_cpu_qlen set to 1000 [ 0.192269] ACPI: Added _OSI(Module Device) [ 0.193019] ACPI: Added _OSI(Processor Device) [ 0.194014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199274] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201570] ACPI: Interpreter enabled [ 0.202071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203016] ACPI: Using IOAPIC for interrupt routing [ 0.204113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205404] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214567] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219203] acpiphp: Slot [2] registered [ 0.220131] acpiphp: Slot [5] registered [ 0.221131] acpiphp: Slot [6] registered [ 0.222133] acpiphp: Slot [7] registered [ 0.223153] acpiphp: Slot [8] registered [ 0.224124] acpiphp: Slot [9] registered [ 0.225124] acpiphp: Slot [10] registered [ 0.226148] acpiphp: Slot [3] registered [ 0.227116] acpiphp: Slot [4] registered [ 0.228169] acpiphp: Slot [11] registered [ 0.229135] acpiphp: Slot [12] registered [ 0.230114] acpiphp: Slot [13] registered [ 0.231184] acpiphp: Slot [14] registered [ 0.232111] acpiphp: Slot [15] registered [ 0.233142] acpiphp: Slot [16] registered [ 0.234178] acpiphp: Slot [17] registered [ 0.235111] acpiphp: Slot [18] registered [ 0.236134] acpiphp: Slot [19] registered [ 0.237128] acpiphp: Slot [20] registered [ 0.238123] acpiphp: Slot [21] registered [ 0.239115] acpiphp: Slot [22] registered [ 0.240116] acpiphp: Slot [23] registered [ 0.241119] acpiphp: Slot [24] registered [ 0.242130] acpiphp: Slot [25] registered [ 0.243128] acpiphp: Slot [26] registered [ 0.244202] acpiphp: Slot [27] registered [ 0.245107] acpiphp: Slot [28] registered [ 0.246110] acpiphp: Slot [29] registered [ 0.247123] acpiphp: Slot [30] registered [ 0.248139] acpiphp: Slot [31] registered [ 0.249083] PCI host bridge to bus 0000:00 [ 0.250026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.254034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.255029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.258073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.260342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.269721] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.270018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.271022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.272019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.276957] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.280048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.283934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306009] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.311017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.318137] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.327023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.425015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438410] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.493517] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550039] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.596731] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645076] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.655017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686197] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.688346] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.691386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.700213] iommu: Default domain type: Passthrough [ 0.702445] SCSI subsystem initialized [ 0.704149] ACPI: bus type USB registered [ 0.706098] usbcore: registered new interface driver usbfs [ 0.708077] usbcore: registered new interface driver hub [ 0.710108] usbcore: registered new device driver usb [ 0.712162] pps_core: LinuxPPS API ver. 1 registered [ 0.714012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718183] PTP clock support registered [ 0.720140] EDAC MC: Ver: 3.0.0 [ 0.723130] PCI: Using ACPI for IRQ routing [ 0.724819] NetLabel: Initializing [ 0.726011] NetLabel: domain hash size = 128 [ 0.728011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730082] NetLabel: unlabeled traffic allowed by default [ 0.733063] vgaarb: loaded [ 0.734388] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743368] clocksource: Switched to clocksource kvm-clock [ 0.863762] VFS: Disk quotas dquot_6.6.0 [ 0.865442] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.868232] *** VALIDATE ramfs *** [ 0.869747] *** VALIDATE hugetlbfs *** [ 0.871442] pnp: PnP ACPI init [ 0.873721] pnp: PnP ACPI: found 6 devices [ 0.891864] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.895348] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898024] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900643] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903666] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906440] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909697] NET: Registered protocol family 2 [ 0.912451] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.919290] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.922770] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928480] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.932728] TCP: Hash tables configured (established 65536 bind 65536) [ 0.936106] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.939631] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942944] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946307] NET: Registered protocol family 1 [ 0.949812] RPC: Registered named UNIX socket transport module. [ 0.952153] RPC: Registered udp transport module. [ 0.953945] RPC: Registered tcp transport module. [ 0.956727] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.959214] NET: Registered protocol family 44 [ 0.961012] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.963360] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.965828] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968701] PCI: CLS 0 bytes, default 64 [ 0.971938] Unpacking initramfs... [ 2.364656] debug: unmapping init [mem 0xffff8c66fcc54000-0xffff8c66fffbffff] [ 2.369925] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373466] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.377568] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.872666] Initialise system trusted keyrings [ 2.874631] Key type blacklist registered [ 2.876624] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.885698] zbud: loaded [ 2.888710] *** VALIDATE nfs *** [ 2.890130] *** VALIDATE nfs4 *** [ 2.891368] pstore: using deflate compression [ 2.894474] Platform Keyring initialized [ 2.996850] NET: Registered protocol family 38 [ 2.999929] Key type asymmetric registered [ 3.002290] Asymmetric key parser 'x509' registered [ 3.005069] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.009856] io scheduler mq-deadline registered [ 3.012599] io scheduler kyber registered [ 3.015353] io scheduler bfq registered [ 3.017750] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.020953] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.024228] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.027361] ACPI: Power Button [PWRF] [ 3.033488] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.040652] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.052580] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.059804] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.081409] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.113357] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.144622] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.150534] Non-volatile memory driver v1.3 [ 3.152365] Linux agpgart interface v0.103 [ 3.185838] virtio_blk virtio1: [vda] 134072 512-byte logical blocks (68.6 MB/65.5 MiB) [ 3.188253] vda: detected capacity change from 0 to 68644864 [ 3.211264] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.213994] vdb: detected capacity change from 0 to 1073741824 [ 3.234711] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.239132] vdc: detected capacity change from 0 to 2621440000 [ 3.266326] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.269372] vdd: detected capacity change from 0 to 2621440000 [ 3.284755] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.287088] vde: detected capacity change from 0 to 4294967296 [ 3.309946] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.313185] vdf: detected capacity change from 0 to 4294967296 [ 3.331137] libphy: Fixed MDIO Bus: probed [ 3.343547] usbcore: registered new interface driver usbserial_generic [ 3.346306] usbserial: USB Serial support registered for generic [ 3.348961] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.352994] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.354681] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.356638] mousedev: PS/2 mouse device common for all mice [ 3.360950] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.363893] rtc_cmos 00:05: RTC can wake from S4 [ 3.367656] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.369198] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.376478] rtc_cmos 00:05: registered as rtc0 [ 3.379221] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.383180] intel_pstate: CPU model not supported [ 3.387341] hid: raw HID events driver (C) Jiri Kosina [ 3.389203] usbcore: registered new interface driver usbhid [ 3.391099] usbhid: USB HID core driver [ 3.392986] drop_monitor: Initializing network drop monitor service [ 3.394853] Initializing XFRM netlink socket [ 3.396578] NET: Registered protocol family 10 [ 3.399265] Segment Routing with IPv6 [ 3.400757] NET: Registered protocol family 17 [ 3.402779] mpls_gso: MPLS GSO support [ 3.408597] RAS: Correctable Errors collector initialized. [ 3.411081] AVX version of gcm_enc/dec engaged. [ 3.412807] AES CTR mode by8 optimization enabled [ 3.491313] sched_clock: Marking stable (3491292762, 0)->(4546735867, -1055443105) [ 3.494372] registered taskstats version 1 [ 3.496043] Loading compiled-in X.509 certificates [ 3.497770] zswap: loaded using pool lzo/zbud [ 3.522779] Key type big_key registered [ 3.537341] Key type encrypted registered [ 3.539337] ima: No TPM chip found, activating TPM-bypass! [ 3.541891] ima: Allocated hash algorithm: sha1 [ 3.543556] ima: No architecture policies found [ 3.545056] evm: Initialising EVM extended attributes: [ 3.546612] evm: security.selinux [ 3.547620] evm: security.ima [ 3.548374] evm: security.capability [ 3.549624] evm: HMAC attrs: 0x1 [ 3.551899] rtc_cmos 00:05: setting system clock to 2026-05-12 19:58:20 UTC (1778615900) [ 3.557888] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 3.560571] debug: unmapping init [mem 0xffffffffab582000-0xffffffffab858fff] [ 3.568075] Write protecting the kernel read-only data: 28672k [ 3.571606] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 3.573908] debug: unmapping init [mem 0xffffffffaa514000-0xffffffffaa5fffff] [ 3.606728] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.615390] systemd[1]: Detected virtualization kvm. [ 3.617038] systemd[1]: Detected architecture x86-64. [ 3.618606] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.646457] systemd[1]: No hostname configured. [ 3.647748] systemd[1]: Set hostname to . [ 3.649353] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651361] systemd[1]: Initializing machine ID from random generator. [ 3.785800] random: systemd: uninitialized urandom read (16 bytes read) [ 3.788383] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.793315] random: systemd: uninitialized urandom read (16 bytes read) [ 3.796446] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.801422] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.330411] device-mapper: uevent: version 1.0.3 [ 4.332658] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.958643] virtio_net virtio0 ens2: renamed from eth0 [ 4.975006] random: fast init done [ 5.012300] scsi host0: ata_piix [ 5.051812] scsi host1: ata_piix [ 5.053891] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.057694] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.484730] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.054043] random: crng init done [ 10.055546] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.438831] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.656222] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.945182] SELinux: Disabled at runtime. [ 12.008490] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.017338] systemd[1]: Detected virtualization kvm. [ 12.019335] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.578590] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.582168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.589979] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.597616] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.600984] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.609144] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.616326] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.660555] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.056082] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.404478] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.414378] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.575397] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.596803] EDAC sbridge: Ver: 1.1.2 [ 15.642621] Key type dns_resolver registered [ 15.976714] NFS: Registering the id_resolver key type [ 15.978804] Key type id_resolver registered [ 15.980591] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg134-server login: [ 43.053893] libcfs: loading out-of-tree module taints kernel. [ 43.070742] Key type ._llcrypt registered [ 43.072645] Key type .llcrypt registered [ 43.127374] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 58.626185] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 60.818231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.842216] alg: No test for adler32 (adler32-zlib) [ 62.657175] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 64.137894] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 66.072494] Key type lgssc registered [ 68.673205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.160915] hrtimer: interrupt took 6933407 ns [ 93.023586] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.863506] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 157.345410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.392169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.839156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.886621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.046727] Lustre: lustre-MDT0000: new disk, initializing [ 159.156093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.183627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.290366] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.929595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.033954] Lustre: 6502:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.072695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.075123] Lustre: Skipped 1 previous similar message [ 182.153973] Lustre: lustre-MDT0001: new disk, initializing [ 182.242471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.308692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.319598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.217703] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.915069] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.570035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.094648] Lustre: lustre-OST0000: new disk, initializing [ 208.106243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.303096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.339669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.368053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.639710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.103449] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.923917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.167848] Lustre: lustre-OST0001: new disk, initializing [ 239.178624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.262523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.322855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.344506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.411887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.287940] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.142804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.987368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.928552] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.492963] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing yml_node [ 289.423503] Lustre: DEBUG MARKER: Client: 2.17.52.126 [ 293.039943] Lustre: DEBUG MARKER: MDS: 2.17.52.126 [ 296.271308] Lustre: DEBUG MARKER: OSS: 2.17.52.126 [ 298.093703] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue May 12 16:03:12 EDT 2026 [ 319.624516] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.816661] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.399670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.429600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.436919] Lustre: Skipped 3 previous similar messages [ 342.498873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.519562] Lustre: Skipped 3 previous similar messages [ 343.949678] Lustre: server umount lustre-MDT0000 complete [ 348.955417] LustreError: 10278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778616245 with bad export cookie 4923962725907940456 [ 348.964539] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.006977] LustreError: 10278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 352.742437] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.748523] LustreError: 6511:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.748814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.755746] Lustre: Skipped 4 previous similar messages [ 352.801418] LustreError: 6511:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 353.771844] LustreError: 6509:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.795260] LustreError: 6509:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 355.339693] Lustre: server umount lustre-MDT0001 complete [ 359.392747] Lustre: server umount lustre-OST0000 complete [ 363.712800] Lustre: server umount lustre-OST0001 complete [ 372.190303] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 382.445953] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 425.517031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.725303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.756473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.835489] Lustre: lustre-MDT0000: new disk, initializing [ 425.964490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.012338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.106337] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 444.553719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.787831] Lustre: 14996:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.896485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.903932] Lustre: Skipped 1 previous similar message [ 445.006453] Lustre: lustre-MDT0001: new disk, initializing [ 445.190907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.250711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.267846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.425134] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 460.119440] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 464.320277] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 473.180795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.557956] Lustre: lustre-OST0000: new disk, initializing [ 473.563577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 473.682312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.533210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.546836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 475.661741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.507656] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 490.359468] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.215438] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.510458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 502.170248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.410889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.346781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.353852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.356564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.356573] Lustre: Skipped 3 previous similar messages [ 506.405720] LustreError: Skipped 1 previous similar message [ 511.235301] Lustre: server umount lustre-OST0000 complete [ 516.590879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.597074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.617789] Lustre: Skipped 2 previous similar messages [ 516.648587] Lustre: Skipped 2 previous similar messages [ 522.213751] Lustre: server umount lustre-MDT0000 complete [ 526.817748] LustreError: 15008:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.852781] LustreError: 15008:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 526.935071] LustreError: 16737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778616423 with bad export cookie 4923962725907941996 [ 526.960602] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.596303] Lustre: server umount lustre-MDT0001 complete [ 534.594847] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:07:09 (1778616429) [ 557.971185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.498502] LustreError: 19468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.603322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.705700] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.379076] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 574.953954] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.988041] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 575.871659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.569180] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 588.802922] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 593.927605] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 596.848760] Lustre: Failing over lustre-MDT0000 [ 596.961568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.973494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.975991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.976018] Lustre: Skipped 2 previous similar messages [ 597.202855] Lustre: server umount lustre-MDT0000 complete [ 601.125819] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.160915] LustreError: 19469:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 617.440453] Lustre: 3654:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778616498/real 1778616498] req@ffff8c6777e93480 x1865014010094080/t0(0) o400->MGC192.168.201.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778616514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.468819] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.040973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.289320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.299976] Lustre: Skipped 1 previous similar message [ 629.394421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 634.338471] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 634.363995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.385449] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.170923] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 644.549670] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 649.697508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.712140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.723494] Lustre: Skipped 1 previous similar message [ 649.733068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.741878] Lustre: Skipped 1 previous similar message [ 655.330945] Lustre: server umount lustre-MDT0000 complete [ 656.866364] LustreError: 20179:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.880368] LustreError: 20179:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 17 previous similar messages [ 659.666557] LustreError: 19448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778616556 with bad export cookie 4923962725907943410 [ 659.677433] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.682488] LustreError: 19448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 659.994820] Lustre: server umount lustre-MDT0001 complete [ 670.659782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.741156] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 686.822532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.079410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 687.089165] Lustre: Skipped 2 previous similar messages [ 692.136833] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 698.808618] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 703.464845] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 706.423996] Lustre: Failing over lustre-MDT0000 [ 706.738474] Lustre: server umount lustre-MDT0000 complete [ 707.579228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 707.602454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.612898] Lustre: Skipped 1 previous similar message [ 722.406246] LustreError: 22467:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.424486] LustreError: 22467:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 724.449096] Lustre: 3655:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778616605/real 1778616605] req@ffff8c66422b9f80 x1865014010123904/t0(0) o400->MGC192.168.201.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778616621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.490756] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.358671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.051244] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 740.322814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 740.336268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 740.354790] Lustre: Skipped 1 previous similar message [ 740.386128] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 741.160800] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 749.888281] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 755.682404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 755.692654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.726707] Lustre: Skipped 1 previous similar message [ 755.734059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.373919] Lustre: server umount lustre-MDT0000 complete [ 766.557302] LustreError: 22451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778616663 with bad export cookie 4923962725907944348 [ 766.558775] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.571230] LustreError: 22451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 767.208492] Lustre: server umount lustre-MDT0001 complete [ 778.561450] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:11:12 (1778616672) [ 797.643732] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 807.603580] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 854.000846] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 867.430958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.690049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 867.725371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 867.783206] Lustre: lustre-MDT0000: new disk, initializing [ 867.849244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.854934] Lustre: Skipped 3 previous similar messages [ 867.875578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 873.441932] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 885.776966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.044144] Lustre: 28456:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 886.545993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 891.364632] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 900.095451] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.347340] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 914.316292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 916.779688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 916.848386] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 916.991807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 917.015228] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 922.935611] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 932.035509] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 938.108492] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 963.042847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.065038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.084919] Lustre: Skipped 1 previous similar message [ 963.095042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.102960] Lustre: Skipped 3 previous similar messages [ 964.373893] Lustre: server umount lustre-OST0000 complete [ 975.939313] Lustre: server umount lustre-MDT0000 complete [ 978.401146] LustreError: 28466:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.415789] LustreError: 28466:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 980.211615] LustreError: 28446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778616877 with bad export cookie 4923962725907944754 [ 980.214149] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.224085] LustreError: 28446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.752275] Lustre: server umount lustre-MDT0001 complete [ 993.884904] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 997.652490] Key type lgssc unregistered [ 998.098665] LNet: 32058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.107483] LNetError: 32058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.145605] LNet: Removed LNI 192.168.201.134@tcp [ 999.202675] Key type .llcrypt unregistered [ 999.207625] Key type ._llcrypt unregistered [ 1025.936166] Key type ._llcrypt registered [ 1025.938524] Key type .llcrypt registered [ 1026.069438] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1027.244269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1027.340721] alg: No test for adler32 (adler32-zlib) [ 1028.467827] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 1028.717488] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1030.424183] Key type lgssc registered [ 1032.044345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1041.678828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1041.711590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.228893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.066800] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1053.685924] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.705771] LustreError: 33456:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1058.130385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.571979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.768883] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1070.774317] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1075.087538] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1082.970464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.270923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.295121] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1088.400050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1090.740334] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1097.943972] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1117.969152] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:16:51 (1778617011) [ 1120.647420] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1122.719638] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1131.506690] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:17:05 (1778617025) [ 1142.469937] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:17:16 (1778617036) [ 1143.813456] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1153.367100] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1153.373085] Lustre: Skipped 2 previous similar messages [ 1155.665354] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:17:29 (1778617049) [ 1157.596537] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1157.600636] Lustre: Skipped 2 previous similar messages [ 1165.920982] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1165.927078] Lustre: Skipped 4 previous similar messages [ 1181.775348] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:17:55 (1778617075) [ 1185.446711] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1190.083516] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1200.186505] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:18:14 (1778617094) [ 1202.392431] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1202.396850] Lustre: Skipped 6 previous similar messages [ 1218.007590] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:18:32 (1778617112) [ 1236.964641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.967568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.984106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1237.009271] Lustre: Skipped 1 previous similar message [ 1242.099823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.110222] Lustre: Skipped 2 previous similar messages [ 1247.227261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.240917] Lustre: Skipped 1 previous similar message [ 1249.248126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1249.388616] Lustre: server umount lustre-OST0000 complete [ 1253.345929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.351050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.372711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.442714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.443914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.730614] Lustre: server umount lustre-MDT0000 complete [ 1262.565387] LustreError: 33455:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.596241] LustreError: 33455:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1264.616495] LustreError: 36679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778617161 with bad export cookie 2176612934978381434 [ 1264.646625] LustreError: 36679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1264.648326] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.030400] Lustre: server umount lustre-MDT0001 complete [ 1276.483036] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1279.244467] Key type lgssc unregistered [ 1279.537335] LNet: 40961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.549554] LNetError: 40961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.564023] LNet: Removed LNI 192.168.201.134@tcp [ 1280.721444] Key type .llcrypt unregistered [ 1280.722909] Key type ._llcrypt unregistered [ 1300.100993] Key type ._llcrypt registered [ 1300.106416] Key type .llcrypt registered [ 1300.211632] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 1317.407976] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1318.588018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.642540] alg: No test for adler32 (adler32-zlib) [ 1319.940876] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 1320.348592] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1322.104195] Key type lgssc registered [ 1323.743535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.605281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.640982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.144533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1368.196074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1368.314738] Lustre: lustre-MDT0000: new disk, initializing [ 1368.376590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.391207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1375.472350] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1387.463818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.646963] Lustre: 44085:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1387.733536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1387.739756] Lustre: Skipped 1 previous similar message [ 1387.878458] Lustre: lustre-MDT0001: new disk, initializing [ 1387.976865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.028677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1388.036788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1393.628327] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1402.240960] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1406.594879] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1415.438224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.856778] Lustre: lustre-OST0000: new disk, initializing [ 1415.865628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1415.993424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.384710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1421.391527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1421.484648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1423.366588] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1432.791430] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.765590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1439.053668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1444.332776] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1444.575894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1446.885548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.891213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.899310] Lustre: Skipped 1 previous similar message [ 1446.920674] Lustre: Skipped 1 previous similar message [ 1452.005280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.013154] Lustre: Skipped 1 previous similar message [ 1452.944489] Lustre: server umount lustre-OST0000 complete [ 1459.681487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1459.693134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.721734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.242035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.249746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.152543] Lustre: server umount lustre-MDT0000 complete [ 1468.758960] LustreError: 44078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778617365 with bad export cookie 16363394195663328227 [ 1468.768456] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.784908] LustreError: 44078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1469.146694] Lustre: server umount lustre-MDT0001 complete [ 1481.591944] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:22:55 (1778617375) [ 1491.172507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.087776] LustreError: 48145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.116349] LustreError: 48145:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1492.271547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.123682] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1497.582956] LustreError: 48146:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.701604] LustreError: 48145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.815254] LustreError: 48146:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.126119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.578843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.799515] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1520.859528] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1526.126445] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1534.152412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1534.384236] LustreError: 49742:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.551408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.857791] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1548.220619] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1552.863929] LustreError: 50336:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1553.993380] Lustre: Modifying parameter general.timeout=1 in log params [ 1555.010597] Lustre: Modifying parameter general.timeout=2 in log params [ 1557.594398] Lustre: Modifying parameter general.timeout=4 in log params [ 1557.605911] Lustre: Skipped 1 previous similar message [ 1561.919797] Lustre: Modifying parameter general.timeout=8 in log params [ 1561.927330] Lustre: Skipped 3 previous similar messages [ 1570.563509] Lustre: Modifying parameter general.timeout=16 in log params [ 1570.570789] Lustre: Skipped 7 previous similar messages [ 1586.859216] Lustre: Modifying parameter general.timeout=32 in log params [ 1586.864392] Lustre: Skipped 15 previous similar messages [ 1620.149703] Lustre: Modifying parameter general.timeout=57 in log params [ 1620.154752] Lustre: Skipped 24 previous similar messages [ 1685.121718] Lustre: Modifying parameter general.timeout=112 in log params [ 1685.134772] Lustre: Skipped 54 previous similar messages [ 1716.063689] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:26:50 (1778617610) [ 1719.574019] Lustre: 57299:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1721.831986] Lustre: 57397:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1725.586454] Lustre: 57613:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1725.593835] Lustre: 57613:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1729.798393] Lustre: 57809:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1729.815622] Lustre: 57809:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1762.058448] Lustre: 59180:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1762.100140] LustreError: 59180:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1768.175950] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:27:42 (1778617662) [ 1785.314556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.331336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.355159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.369336] Lustre: Skipped 1 previous similar message [ 1791.214476] Lustre: server umount lustre-MDT0000 complete [ 1791.460748] LustreError: 54256:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.508327] LustreError: 54256:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1796.880554] LustreError: 48864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778617693 with bad export cookie 16363394195663329109 [ 1796.890225] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.900205] LustreError: 48864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1797.223325] Lustre: server umount lustre-MDT0001 complete [ 1813.061678] Lustre: server umount lustre-OST0000 complete [ 1845.471953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.017526] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1846.076923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1846.094915] Lustre: Skipped 40 previous similar messages [ 1846.467040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.505892] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1864.060695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.268639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1864.326685] Lustre: 61713:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1870.064744] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1876.751693] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1881.654230] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1890.576513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1890.877800] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1891.215627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.223969] Lustre: Skipped 1 previous similar message [ 1899.404134] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1907.181825] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1909.754669] Lustre: 63943:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1920.482078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.488769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.495888] LustreError: Skipped 1 previous similar message [ 1920.497893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.497900] Lustre: Skipped 3 previous similar messages [ 1920.519657] Lustre: Skipped 3 previous similar messages [ 1925.683678] Lustre: server umount lustre-OST0000 complete [ 1930.727981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.751142] Lustre: Skipped 1 previous similar message [ 1937.403449] Lustre: server umount lustre-MDT0000 complete [ 1940.967965] LustreError: 61725:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.017207] LustreError: 61725:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1942.032450] LustreError: 61707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778617838 with bad export cookie 16363394195663332378 [ 1942.045415] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.371785] Lustre: server umount lustre-MDT0001 complete [ 1955.679129] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1958.989945] Key type lgssc unregistered [ 1959.270887] LNet: 65139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.276383] LNetError: 65139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.301646] LNet: Removed LNI 192.168.201.134@tcp [ 1960.394579] Key type .llcrypt unregistered [ 1960.398081] Key type ._llcrypt unregistered [ 1987.024988] Key type ._llcrypt registered [ 1987.031759] Key type .llcrypt registered [ 1987.143377] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1988.509056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.574565] alg: No test for adler32 (adler32-zlib) [ 1989.619550] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 1989.935815] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1991.608203] Key type lgssc registered [ 1993.019425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.925844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2001.963583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.604046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.096047] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2013.157250] LustreError: 66538:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.189852] LustreError: 66538:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2018.290156] LustreError: 66537:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.884276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.873546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2026.512821] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2034.080576] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2039.208260] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2047.649768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.595660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.812487] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2064.528755] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2073.317626] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2086.884527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2086.904295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.937736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.232410] Lustre: server umount lustre-MDT0000 complete [ 2092.514586] LustreError: 66538:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.537654] LustreError: 66538:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2094.842785] LustreError: 66517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778617991 with bad export cookie 254068465617401335 [ 2094.853468] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.861175] LustreError: 66517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2095.182956] Lustre: server umount lustre-MDT0001 complete [ 2110.242828] Lustre: server umount lustre-OST0000 complete [ 2142.546524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.192157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2143.232570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2143.633467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.535883] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2161.455513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.651734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2161.694689] Lustre: 71072:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2162.366286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.399356] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2173.466648] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2178.045967] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2186.729864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.092984] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2187.612856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.188781] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2201.223412] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2203.583973] Lustre: 73308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.003171] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2211.811074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.830497] Lustre: Skipped 2 previous similar messages [ 2211.833548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.845162] Lustre: Skipped 3 previous similar messages [ 2216.420726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.436084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.472707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2217.892837] Lustre: server umount lustre-OST0000 complete [ 2223.585088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2223.590621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.602960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.609165] Lustre: Skipped 1 previous similar message [ 2232.291896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.301496] Lustre: Skipped 3 previous similar messages [ 2237.410790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2237.826040] Lustre: server umount lustre-MDT0000 complete [ 2242.393897] LustreError: 71110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778618139 with bad export cookie 254068465617402182 [ 2242.397548] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.402791] LustreError: 71110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2242.899709] Lustre: server umount lustre-MDT0001 complete [ 2257.770976] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2262.561116] Key type lgssc unregistered [ 2263.091892] LNet: 74504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.124501] LNetError: 74504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.155950] LNet: Removed LNI 192.168.201.134@tcp [ 2264.406439] Key type .llcrypt unregistered [ 2264.408146] Key type ._llcrypt unregistered [ 2297.162452] Key type ._llcrypt registered [ 2297.165828] Key type .llcrypt registered [ 2297.365543] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2299.228701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.459039] alg: No test for adler32 (adler32-zlib) [ 2300.671414] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 2301.016705] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2302.752300] Key type lgssc registered [ 2304.191230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.703099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2312.722890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.542377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.733175] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2324.963245] LustreError: 75903:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.989718] LustreError: 75903:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2330.092434] LustreError: 75904:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.346230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.071755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.216479] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2343.288355] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2348.830833] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2357.555269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.379380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.723083] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2374.336488] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2381.424539] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2382.763543] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2384.085324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2386.531491] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2386.535923] Lustre: Skipped 1 previous similar message [ 2391.063178] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2391.069850] Lustre: Skipped 3 previous similar messages [ 2399.529155] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2399.534859] Lustre: Skipped 6 previous similar messages [ 2416.258070] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2416.263549] Lustre: Skipped 14 previous similar messages [ 2449.519925] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2449.534920] Lustre: Skipped 26 previous similar messages [ 2515.279709] Lustre: 83685:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2516.474844] Lustre: Disabling parameter general.jobid_var= in log params [ 2516.479745] Lustre: Skipped 42 previous similar messages [ 2518.826625] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:40:12 (1778618412) [ 2520.931743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2523.079631] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:40:17 (1778618417) [ 2528.750683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.763555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.790694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.769549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.773989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.778392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.827194] Lustre: Skipped 1 previous similar message [ 2534.655524] Lustre: server umount lustre-OST0000 complete [ 2540.001754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.007624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.024744] Lustre: Skipped 1 previous similar message [ 2540.044102] Lustre: Skipped 1 previous similar message [ 2545.123858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.163119] Lustre: server umount lustre-MDT0000 complete [ 2549.222059] LustreError: 75904:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.260699] LustreError: 75904:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2550.968334] LustreError: 75884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778618447 with bad export cookie 15570778595576077384 [ 2550.981509] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.988789] LustreError: 75884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2551.591806] Lustre: server umount lustre-MDT0001 complete [ 2564.747634] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2568.317508] Key type lgssc unregistered [ 2568.729476] LNet: 84918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.740574] LNetError: 84918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.766850] LNet: Removed LNI 192.168.201.134@tcp [ 2570.819674] Key type .llcrypt unregistered [ 2570.825263] Key type ._llcrypt unregistered [ 2576.654319] Key type ._llcrypt registered [ 2576.656185] Key type .llcrypt registered [ 2576.787802] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2582.658933] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules [ 2584.747873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2584.835842] alg: No test for adler32 (adler32-zlib) [ 2586.119780] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 2586.546899] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2588.329495] Key type lgssc registered [ 2590.438545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2592.376457] LustreError: 86136:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2615.178185] LustreError: 86136:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2615.311103] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2626.638868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2626.685688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.557815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.378271] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2638.827058] LustreError: 87141:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.861676] LustreError: 87141:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2643.950524] LustreError: 87142:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.868571] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:42:20 (1778618540) [ 2649.080849] LustreError: 87141:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.167034] LustreError: 87142:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.496483] LustreError: 87173:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2653.515659] LustreError: 87173:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2653.530247] LustreError: 87173:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2654.504319] Lustre: server umount lustre-MDT0000 complete [ 2673.420928] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2677.002476] Key type lgssc unregistered [ 2677.379630] LNet: 88636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.391312] LNetError: 88636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.440726] LNet: Removed LNI 192.168.201.134@tcp [ 2678.399707] Key type .llcrypt unregistered [ 2678.402604] Key type ._llcrypt unregistered [ 2699.799678] Key type ._llcrypt registered [ 2699.807773] Key type .llcrypt registered [ 2699.963635] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2701.922642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2702.021384] alg: No test for adler32 (adler32-zlib) [ 2703.234615] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 2703.500447] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2705.192944] Key type lgssc registered [ 2706.902127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2717.096966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2717.123169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.922181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.638990] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2728.439315] LustreError: 89843:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.477950] LustreError: 89843:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2733.540206] LustreError: 89844:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.178612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.032562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.370192] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2750.279932] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2754.471983] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2762.442664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.100270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.884602] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2777.621540] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2840.981895] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:45:35 (1778618735) [ 2843.836696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2846.163709] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:45:40 (1778618740) [ 2854.374206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.400701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.444803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.428983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.429175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.445567] Lustre: Skipped 1 previous similar message [ 2856.453187] Lustre: Skipped 1 previous similar message [ 2858.038323] Lustre: server umount lustre-MDT0000 complete [ 2861.547511] LustreError: 90559:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.566992] LustreError: 90559:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2862.789111] LustreError: 89825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778618759 with bad export cookie 14569338261114949280 [ 2862.790653] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.798856] LustreError: 89825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2863.282707] Lustre: server umount lustre-MDT0001 complete [ 2879.126059] Lustre: server umount lustre-OST0000 complete [ 2893.622409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.324955] LustreError: 93809:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.646040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.946975] LustreError: 93810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.977979] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2910.210650] LustreError: 93809:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.225660] LustreError: 93809:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2911.164875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.034727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.424617] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2926.884886] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2930.990555] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2945.943568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.271200] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2946.280057] LustreError: 93798:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2946.349508] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2946.359359] LustreError: 95668:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2946.366633] LustreError: 95668:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2946.370464] LustreError: 95668:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2946.455933] Lustre: server umount lustre-OST0000 complete [ 2946.460359] LustreError: 95668:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2956.503718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.663265] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2956.669116] LustreError: 93798:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2956.688618] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2956.693978] LustreError: 96133:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2956.698889] LustreError: 96133:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2956.705364] LustreError: 96133:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2956.758497] Lustre: server umount lustre-OST0000 complete [ 2956.762471] LustreError: 96133:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2967.260167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.518290] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2967.524300] Lustre: Found index 0 for lustre-OST0000, updating log [ 2967.531579] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2967.615759] Lustre: lustre-OST0000: new disk, initializing [ 2967.619663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2967.987915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.176906] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2979.589168] Lustre: Failing over lustre-OST0000 [ 2979.765849] Lustre: server umount lustre-OST0000 complete [ 2986.996927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.007028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.023346] Lustre: Skipped 1 previous similar message [ 2987.048916] Lustre: Skipped 2 previous similar messages [ 2990.450328] Lustre: server umount lustre-MDT0000 complete [ 2992.097456] LustreError: 93810:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.107079] LustreError: 93810:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2994.617632] LustreError: 93789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778618891 with bad export cookie 14569338261114952017 [ 2994.631289] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2994.644370] LustreError: 93789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2995.016554] Lustre: server umount lustre-MDT0001 complete [ 3004.585465] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:48:18 (1778618898) [ 3024.198515] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3039.200775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.962135] LustreError: 98981:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.105975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.818291] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3061.804160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.725269] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3072.046349] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3081.436808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.446643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.457333] Lustre: Skipped 1 previous similar message [ 3090.580347] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3101.089696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.268598] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3101.384959] Lustre: lustre-OST0001: new disk, initializing [ 3101.390231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3107.130714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3107.147608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3107.360945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3110.764174] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3122.939454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3133.457319] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3140.927395] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 3141.096292] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3141.106564] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.009236] LustreError: 100902:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.032023] LustreError: 100902:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 3146.200527] Lustre: server umount lustre-MDT0001 complete [ 3160.455622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.796766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3160.816386] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3160.924041] Lustre: 98969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3162.269720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3162.280110] Lustre: Skipped 1 previous similar message [ 3162.998529] LustreError: 102772:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3163.007050] LustreError: 102772:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3163.018815] LustreError: 98973:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3163.230518] LustreError: 102779:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.134@tcp: cfg command failed: rc = -17 [ 3163.246895] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3163.246895] [ 3167.729720] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3168.426744] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3180.515710] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 16:51:14 (1778619074) [ 3202.084756] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3224.402302] Lustre: 104820:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3251.428793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3256.120369] Lustre: 105954:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3354.932785] Lustre: server umount lustre-MDT0000 complete [ 3356.144060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3356.152307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.182846] Lustre: Skipped 2 previous similar messages [ 3356.195103] LustreError: 99696:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.221814] LustreError: 99696:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 3359.266322] LustreError: 101251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778619256 with bad export cookie 14569338261114953018 [ 3359.266743] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.292223] LustreError: 101251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3359.930938] Lustre: server umount lustre-MDT0001 complete [ 3375.317043] Lustre: server umount lustre-OST0000 complete [ 3378.528270] Lustre: 89154:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778619259/real 1778619259] req@ffff8c664d083b80 x1865016777509504/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778619275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3378.570414] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.592305] Lustre: Skipped 3 previous similar messages [ 3380.706095] Lustre: 89157:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778619261/real 1778619261] req@ffff8c664d081180 x1865016777509760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778619277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3381.424255] Lustre: server umount lustre-OST0001 complete [ 3388.313199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3400.862576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3412.577983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3426.453194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3450.678778] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3464.735214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.787541] Lustre: lustre-MDT0000: reset Object Index mappings [ 3465.772354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.055385] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3483.920939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.959603] Lustre: lustre-MDT0001: reset Object Index mappings [ 3491.166640] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3495.000660] Lustre: 111007:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3502.399731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.599979] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3511.674096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3511.686640] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3522.240328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.197652] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3528.197861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3530.496446] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3538.991963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3543.757977] Lustre: 112852:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3602.473934] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:58:16 (1778619496) [ 3630.572674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.578272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.591371] Lustre: Skipped 2 previous similar messages [ 3630.600920] Lustre: Skipped 7 previous similar messages [ 3634.952332] Lustre: server umount lustre-MDT0000 complete [ 3635.686448] LustreError: 110858:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.708563] LustreError: 110858:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 18 previous similar messages [ 3639.351413] LustreError: 109872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778619536 with bad export cookie 14569338261114999904 [ 3639.353738] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.356971] LustreError: 109872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3640.804055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3640.811883] Lustre: Skipped 1 previous similar message [ 3646.062435] Lustre: server umount lustre-MDT0001 complete [ 3665.376144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3665.716480] Lustre: server umount lustre-OST0000 complete [ 3676.836430] Lustre: server umount lustre-OST0001 complete [ 3686.140175] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 3699.290761] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3747.267032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3747.569484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3747.624957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3747.738666] Lustre: lustre-MDT0000: new disk, initializing [ 3747.864056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.870971] Lustre: Skipped 3 previous similar messages [ 3747.887949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3755.083802] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3765.822868] Lustre: server umount lustre-MDT0000 complete [ 3779.849700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.520824] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3780.539450] Lustre: 117086:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3780.570482] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3786.489596] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3792.372128] Lustre: server umount lustre-MDT0000 complete [ 3803.429752] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:01:37 (1778619697) [ 3805.136972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3807.948290] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:01:41 (1778619701) [ 3831.722097] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3845.735555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.868611] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3866.243592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.466686] Lustre: 119105:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.561580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.566414] Lustre: Skipped 1 previous similar message [ 3866.743051] Lustre: lustre-MDT0001: new disk, initializing [ 3867.012813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.026149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3873.172117] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3879.863243] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3890.151894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.632035] Lustre: lustre-OST0000: new disk, initializing [ 3890.641683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3891.933020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3891.956798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3892.134321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3898.697748] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3912.751299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3912.976245] Lustre: lustre-OST0001: new disk, initializing [ 3912.980471] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3914.506188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3914.520659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3914.727162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3921.530393] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3932.524296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3936.567346] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4428.587921] Lustre: 127920:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4428.604082] Lustre: 127920:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4428.619425] Lustre: 127920:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4428.628194] Lustre: 127920:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4428.641119] Lustre: 127920:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4428.652953] Lustre: 127920:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4758.768671] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:17:32 (1778620652) [ 4774.887555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.894342] Lustre: Skipped 4 previous similar messages [ 4774.899346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4774.904900] Lustre: Skipped 5 previous similar messages [ 4780.012443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4780.021874] Lustre: Skipped 3 previous similar messages [ 4781.014569] Lustre: server umount lustre-MDT0000 complete [ 4785.141762] LustreError: 122291:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4785.169281] LustreError: 122291:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4785.593317] LustreError: 127980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778620682 with bad export cookie 14569338261115069792 [ 4785.595760] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.604267] LustreError: 127980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4790.264431] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4790.295487] Lustre: Skipped 3 previous similar messages [ 4790.296186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4792.353941] Lustre: server umount lustre-MDT0001 complete [ 4804.694868] Lustre: server umount lustre-OST0000 complete [ 4816.519487] Lustre: server umount lustre-OST0001 complete [ 4825.628590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4837.200350] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4886.403836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.731670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4886.776918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4886.880835] Lustre: lustre-MDT0000: new disk, initializing [ 4886.997735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.008120] Lustre: Skipped 5 previous similar messages [ 4887.044859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4892.101402] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4906.041733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.281532] Lustre: 130950:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.408858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4906.418883] Lustre: Skipped 1 previous similar message [ 4906.614777] Lustre: lustre-MDT0001: new disk, initializing [ 4906.828883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4906.842368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4912.446509] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4921.432194] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4926.216157] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4933.996932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4934.411417] Lustre: lustre-OST0000: new disk, initializing [ 4934.421314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4935.567540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4935.587947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4935.739962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4941.324265] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4949.186678] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4960.302403] Lustre: lustre-MDD0000: changelog on [ 4963.411330] Lustre: lustre-MDD0001: changelog on [ 7096.739572] Lustre: lustre-MDD0001: changelog off [ 7099.394671] Lustre: 130958:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7100.636698] Lustre: lustre-MDD0000: changelog off [ 7111.650447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7111.659962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.686761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.688948] Lustre: Skipped 1 previous similar message [ 7112.693612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.699459] Lustre: Skipped 1 previous similar message [ 7117.803656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.826823] Lustre: Skipped 1 previous similar message [ 7122.404806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7122.805831] Lustre: server umount lustre-OST0000 complete [ 7128.033588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.037448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.058228] Lustre: Skipped 2 previous similar messages [ 7128.070683] Lustre: Skipped 1 previous similar message [ 7133.826332] Lustre: server umount lustre-MDT0000 complete [ 7138.198455] LustreError: 130943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778623035 with bad export cookie 14569338261115898858 [ 7138.210097] LustreError: 130943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7138.213388] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.650530] Lustre: server umount lustre-MDT0001 complete [ 7149.757513] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 7152.719696] Key type lgssc unregistered [ 7153.034870] LNet: 141734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7153.039909] LNetError: 141734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7153.049813] LNet: Removed LNI 192.168.201.134@tcp [ 7153.734263] Key type .llcrypt unregistered [ 7153.737107] Key type ._llcrypt unregistered [ 7181.214384] Key type ._llcrypt registered [ 7181.216291] Key type .llcrypt registered [ 7181.367848] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 7200.435382] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7202.294613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7202.337031] alg: No test for adler32 (adler32-zlib) [ 7203.614073] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 7203.734381] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 7205.376644] Key type lgssc registered [ 7207.114197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7256.556672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7256.632028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.198815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7258.285974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7258.471270] Lustre: lustre-MDT0000: new disk, initializing [ 7258.686719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.743035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7264.584497] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7278.873897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.113800] Lustre: 144909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7279.185410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7279.189294] Lustre: Skipped 1 previous similar message [ 7279.269804] Lustre: lustre-MDT0001: new disk, initializing [ 7279.460994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7279.575908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7279.596287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7285.505416] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7295.227937] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7301.345223] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7311.799633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.439854] Lustre: lustre-OST0000: new disk, initializing [ 7312.448628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.591800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.211642] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7321.738441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7321.756778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7321.990225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7331.635112] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7339.680626] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_module obdecho/obdecho [ 7342.038464] LustreError: 147523:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7342.050325] LustreError: 147523:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7344.854291] LustreError: 147620:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7361.512978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7361.537648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7361.559661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.043861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.051852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.074689] Lustre: Skipped 1 previous similar message [ 7363.102863] Lustre: Skipped 2 previous similar messages [ 7366.870799] Lustre: server umount lustre-MDT0000 complete [ 7371.921268] LustreError: 144900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778623268 with bad export cookie 13590840949313340590 [ 7371.931728] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7371.933622] LustreError: 144900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7372.791826] Lustre: server umount lustre-MDT0001 complete [ 7389.156523] Lustre: server umount lustre-OST0000 complete [ 7420.287689] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 7433.899970] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7484.146835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.438608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7484.493191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7484.643796] Lustre: lustre-MDT0000: new disk, initializing [ 7484.782780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.811981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7490.961660] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7504.636491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.793572] Lustre: 151321:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7504.858437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7504.862370] Lustre: Skipped 1 previous similar message [ 7504.931598] Lustre: lustre-MDT0001: new disk, initializing [ 7505.038667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7505.080259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7505.111487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7510.493028] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7520.335487] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7524.129197] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7532.300445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.712246] Lustre: lustre-OST0000: new disk, initializing [ 7532.717305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7532.837267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7538.288108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7538.300617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7538.461653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7541.819118] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7552.258708] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7558.212574] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7558.582445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7564.611548] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7564.862414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7569.390615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7569.395454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.415812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7569.447394] Lustre: Skipped 1 previous similar message [ 7573.453200] Lustre: server umount lustre-OST0000 complete [ 7579.618761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.651666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7579.667236] Lustre: Skipped 2 previous similar messages [ 7584.703137] Lustre: server umount lustre-MDT0000 complete [ 7584.738583] LustreError: 151332:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.758605] LustreError: 151332:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 7589.421956] LustreError: 151313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778623486 with bad export cookie 13590840949313341549 [ 7589.429264] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7589.434705] LustreError: 151313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7589.841203] Lustre: server umount lustre-MDT0001 complete [ 7594.421524] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:04:48 (1778623488) [ 7597.143884] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7600.269927] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:04:53 (1778623493) [ 7612.923716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.879496] LustreError: 155330:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.070054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.051771] LustreError: 155331:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7619.095936] LustreError: 155331:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7619.932975] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7623.138185] LustreError: 155330:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.257916] LustreError: 155331:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.311341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.919248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7644.998289] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7650.491603] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7660.553138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.894196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.897330] Lustre: Skipped 1 previous similar message [ 7669.316228] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7675.478841] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7693.836410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.855078] Lustre: Skipped 2 previous similar messages [ 7693.867790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.882103] Lustre: Skipped 1 previous similar message [ 7698.402639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7698.419445] LustreError: Skipped 1 previous similar message [ 7698.446432] LustreError: 156042:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.571690] Lustre: server umount lustre-MDT0000 complete [ 7704.068826] LustreError: 155310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778623601 with bad export cookie 13590840949313342431 [ 7704.099472] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7704.106872] LustreError: 155310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7706.941581] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 7706.955920] Lustre: Skipped 1 previous similar message [ 7709.168106] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.184894] Lustre: Skipped 1 previous similar message [ 7710.593670] Lustre: server umount lustre-MDT0001 complete [ 7719.480743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7735.264696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.623808] LustreError: 158859:0:(llog.c:1646:llog_backup()) MGC192.168.201.134@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7735.634703] Lustre: 158859:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7737.741393] LustreError: 158865:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.783289] LustreError: 158865:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 7770.599158] LustreError: 158866:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.635724] LustreError: 158866:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 23 previous similar messages [ 7796.192426] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7796.198971] Lustre: 158859:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.201.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7835.617607] LustreError: 158864:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.648740] LustreError: 158864:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 41 previous similar messages [ 7837.172390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.276953] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7843.283513] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7855.823698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.827511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7875.110873] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc9c5dd606a7a538 [ 7875.127843] Lustre: MGC192.168.201.134@tcp: Connection restored to 0@lo (at 0@lo) [ 7880.703789] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7882.038522] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7891.565947] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7894.748402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7901.312809] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7903.973757] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7931.877967] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.878971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7931.879364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.879369] Lustre: Skipped 1 previous similar message [ 7931.893361] Lustre: Skipped 1 previous similar message [ 7932.560714] Lustre: server umount lustre-MDT0000 complete [ 7937.522066] LustreError: 155311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778623834 with bad export cookie 13590840949313348920 [ 7937.527602] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.537099] LustreError: 155311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7938.139310] Lustre: server umount lustre-MDT0001 complete [ 7954.330225] Lustre: server umount lustre-OST0000 complete [ 7984.136372] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 7997.857561] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8048.903628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.379403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8049.409391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8049.566330] Lustre: lustre-MDT0000: new disk, initializing [ 8049.698721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.705941] Lustre: Skipped 1 previous similar message [ 8049.732508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8056.091544] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8071.566185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.756268] Lustre: 163717:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8072.148394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8077.834352] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8087.135639] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8092.691491] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8101.981677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8102.372192] Lustre: lustre-OST0000: new disk, initializing [ 8102.377201] Lustre: Skipped 1 previous similar message [ 8102.384267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8102.392164] Lustre: Skipped 2 previous similar messages [ 8108.179892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8108.194065] Lustre: Skipped 1 previous similar message [ 8108.198471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8108.365720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.010629] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8121.687073] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8129.144485] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8129.522129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8136.032730] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8136.499902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8144.359651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.360384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.361873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.361884] Lustre: Skipped 2 previous similar messages [ 8144.379041] LustreError: Skipped 1 previous similar message [ 8144.438185] Lustre: Skipped 1 previous similar message [ 8145.720967] Lustre: server umount lustre-OST0000 complete [ 8158.064405] Lustre: server umount lustre-MDT0000 complete [ 8159.716432] LustreError: 163729:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.753502] LustreError: 163729:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 39 previous similar messages [ 8164.212249] LustreError: 165461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624061 with bad export cookie 13590840949313365720 [ 8164.222707] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8164.235388] LustreError: 165461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8164.726815] Lustre: server umount lustre-MDT0001 complete [ 8177.924220] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:14:31 (1778624071) [ 8185.976889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8195.015238] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8206.558132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.057813] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8218.752462] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8224.146705] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8233.188778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8242.367724] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8249.972529] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8274.906493] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:16:08 (1778624168) [ 8282.084113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.101388] LustreError: Skipped 1 previous similar message [ 8282.102930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.111936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8282.126580] Lustre: Skipped 3 previous similar messages [ 8282.140757] Lustre: Skipped 4 previous similar messages [ 8287.054934] Lustre: server umount lustre-OST0000 complete [ 8297.984576] Lustre: server umount lustre-MDT0000 complete [ 8302.389432] LustreError: 167709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624199 with bad export cookie 13590840949313366602 [ 8302.402600] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.858852] Lustre: server umount lustre-MDT0001 complete [ 8316.882456] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 8320.348803] Key type lgssc unregistered [ 8320.738519] LNet: 171128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8320.753711] LNetError: 171128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8320.770725] LNet: Removed LNI 192.168.201.134@tcp [ 8321.941723] Key type .llcrypt unregistered [ 8321.943471] Key type ._llcrypt unregistered [ 8344.447567] Key type ._llcrypt registered [ 8344.450120] Key type .llcrypt registered [ 8344.532894] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8346.578514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8346.591870] alg: No test for adler32 (adler32-zlib) [ 8347.805384] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 8348.163411] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 8349.936299] Key type lgssc registered [ 8351.373585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8361.901273] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8361.929885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.633094] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8382.753077] LustreError: 172357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.134@tcp: failed processing log, type 4: rc = -110 [ 8387.873053] LustreError: 172357:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8387.892372] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8387.923740] LustreError: MGC192.168.201.134@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8387.935694] LustreError: 172357:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8387.945912] LustreError: 172357:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 8387.959268] LustreError: 172357:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8388.083817] Lustre: server umount lustre-OST0000 complete [ 8388.095779] LustreError: 172357:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8399.186417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.742954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.490889] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8410.100080] LustreError: 172857:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.126340] LustreError: 172857:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8415.202969] LustreError: 172858:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.612702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.534298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8426.216253] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8433.693839] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8440.190674] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8448.700668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.011550] LustreError: 174447:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8449.027757] Lustre: 174447:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.201.134@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8449.308914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.117217] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8463.163989] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8466.483097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.509093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.510855] Lustre: Skipped 2 previous similar messages [ 8466.542045] Lustre: Skipped 2 previous similar messages [ 8471.523814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8471.534388] Lustre: Skipped 2 previous similar messages [ 8471.947379] Lustre: server umount lustre-MDT0000 complete [ 8476.650318] LustreError: 172857:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.651418] LustreError: 172839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624373 with bad export cookie 11337681378038416493 [ 8476.684111] LustreError: 172857:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8476.697493] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8476.706530] LustreError: 172839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8476.832228] LustreError: 173570:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7238248 x1865022696354176/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8477.048384] Lustre: server umount lustre-MDT0001 complete [ 8492.068785] Lustre: server umount lustre-OST0000 complete [ 8503.372399] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 8515.916361] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8561.558348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.901516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8561.975462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8562.130224] Lustre: lustre-MDT0000: new disk, initializing [ 8562.259471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.284722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8567.231956] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8578.956590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.085501] Lustre: 177826:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8579.196587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8579.201593] Lustre: Skipped 1 previous similar message [ 8579.270203] Lustre: lustre-MDT0001: new disk, initializing [ 8579.367493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8579.422148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8579.430640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8584.477897] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8592.275415] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8595.799594] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8603.474047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8603.856848] Lustre: lustre-OST0000: new disk, initializing [ 8603.860737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8604.020436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.322853] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8613.028538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8613.041072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8613.145111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8619.159169] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8624.144523] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8624.362991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8628.674404] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8628.849961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8633.826950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8633.843705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.863703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8638.987836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8638.992423] Lustre: Skipped 1 previous similar message [ 8644.068487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8644.072804] Lustre: Skipped 1 previous similar message [ 8645.090117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8645.231688] Lustre: server umount lustre-OST0000 complete [ 8651.233212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8651.240081] LustreError: Skipped 1 previous similar message [ 8651.248207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.268649] Lustre: Skipped 1 previous similar message [ 8654.306077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8654.306677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.330044] Lustre: Skipped 2 previous similar messages [ 8655.791133] Lustre: server umount lustre-MDT0000 complete [ 8659.427782] LustreError: 177832:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8659.445452] LustreError: 177832:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8660.148949] LustreError: 177817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624557 with bad export cookie 11337681378038417207 [ 8660.165767] LustreError: 177817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8660.181922] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.599368] Lustre: server umount lustre-MDT0001 complete [ 8669.857259] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:22:44 (1778624564) [ 8689.919543] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 8692.940649] Key type lgssc unregistered [ 8693.361704] LNet: 182348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.368918] LNetError: 182348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8694.439495] LNet: Removed LNI 192.168.201.134@tcp [ 8695.378741] Key type .llcrypt unregistered [ 8695.382626] Key type ._llcrypt unregistered [ 8716.331838] Key type ._llcrypt registered [ 8716.334122] Key type .llcrypt registered [ 8716.467855] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8717.899229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8718.053084] alg: No test for adler32 (adler32-zlib) [ 8719.140622] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 8719.497560] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 8721.200187] Key type lgssc registered [ 8722.434384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8731.108572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8731.152101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.727599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.968926] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8742.895475] LustreError: 183550:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8742.923132] LustreError: 183550:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8747.928784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.007727] LustreError: 183551:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.565619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8753.652652] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8760.754382] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8765.468587] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8773.929952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8774.056756] LustreError: 185193:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.156842] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8774.159204] LustreError: 185189:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8774.162606] Lustre: 185189:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.201.134@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8774.242559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.421577] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8787.286587] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8793.062836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.067895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.074642] Lustre: Skipped 1 previous similar message [ 8798.179894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.189808] Lustre: Skipped 4 previous similar messages [ 8803.300985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.320332] Lustre: Skipped 2 previous similar messages [ 8804.320305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8804.752728] Lustre: server umount lustre-MDT0000 complete [ 8808.415121] LustreError: 185839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624705 with bad export cookie 13212964976661415203 [ 8808.421276] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8808.424868] LustreError: 185839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8808.444633] LustreError: 183545:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.452941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8808.481639] Lustre: Skipped 1 previous similar message [ 8808.495410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8808.708987] Lustre: server umount lustre-MDT0001 complete [ 8814.588783] Lustre: server umount lustre-OST0000 complete [ 8825.976641] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 8838.739250] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8897.548537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.891840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8897.944635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8898.074619] Lustre: lustre-MDT0000: new disk, initializing [ 8898.194976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.247224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8904.186689] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8916.034187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8916.280688] Lustre: 188573:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8916.365051] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8916.367323] Lustre: Skipped 1 previous similar message [ 8916.370631] LustreError: 189338:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8916.381964] LustreError: 189338:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8916.389166] Lustre: 189338:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8916.412445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8916.417567] Lustre: Skipped 1 previous similar message [ 8916.510415] Lustre: lustre-MDT0001: new disk, initializing [ 8916.639168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8916.682953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8916.693592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8921.831154] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8934.667177] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8939.878951] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8947.828304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8948.178799] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8948.188691] Lustre: Skipped 1 previous similar message [ 8948.197336] LustreError: 190289:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8948.219206] LustreError: 190289:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8948.229766] Lustre: 190289:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8948.245665] Lustre: 190289:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8948.269419] Lustre: lustre-OST0000: new disk, initializing [ 8948.277497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8948.396886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.431820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8954.456480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8954.584944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8955.140796] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8963.391571] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8968.806692] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8968.967022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8974.655419] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8974.928617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8980.451061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8980.454473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.478133] LustreError: Skipped 1 previous similar message [ 8980.485370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8980.493586] Lustre: Skipped 1 previous similar message [ 8980.527609] Lustre: Skipped 1 previous similar message [ 8983.669044] Lustre: server umount lustre-OST0000 complete [ 8988.642513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8988.651416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.676519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8994.490513] Lustre: server umount lustre-MDT0000 complete [ 8995.809977] LustreError: 188584:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.834136] LustreError: 188584:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8998.974755] LustreError: 188564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778624895 with bad export cookie 13212964976661415924 [ 8998.978791] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8998.991988] LustreError: 188564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8999.542015] Lustre: server umount lustre-MDT0001 complete [ 9011.251325] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:28:24 (1778624904) [ 9037.643630] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9051.713947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.132751] LustreError: 193302:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9052.173447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9057.630106] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9067.281797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.759958] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9076.274305] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9084.736430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9085.226349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.235515] Lustre: Skipped 1 previous similar message [ 9092.701728] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9103.301910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9103.508354] Lustre: lustre-OST0001: new disk, initializing [ 9103.511462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9110.119556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9110.150607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9110.310308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9113.315659] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9126.914081] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9138.355755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9140.826675] Lustre: DEBUG MARKER: ADD OST3 [ 9146.770159] loop: module loaded [ 9146.817219] loop0: detected capacity change from 0 to 204800000 [ 9147.024197] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9147.049601] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9147.059696] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9154.440684] Lustre: DEBUG MARKER: STOP OST3 [ 9154.936964] loop0: detected capacity change from 0 to 204800000 [ 9162.317226] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9162.554355] Lustre: lustre-OST0003: new disk, initializing [ 9162.565339] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9162.734578] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9162.740683] Lustre: Skipped 1 previous similar message [ 9164.107125] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9164.276209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9164.282345] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9164.338155] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9165.849135] Lustre: Failing over lustre-OST0003 [ 9165.985643] Lustre: server umount lustre-OST0003 complete [ 9166.179787] LustreError: 197721:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9169.201265] LustreError: 194755:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.229273] LustreError: 194755:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9169.399720] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.402451] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9169.416190] Lustre: Skipped 1 previous similar message [ 9170.360796] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9182.708479] Lustre: DEBUG MARKER: START OST3 again [ 9190.150700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9190.471967] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9191.789397] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9192.305282] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9192.305578] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9196.971683] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9218.421434] Lustre: Failing over lustre-OST0003 [ 9218.509735] Lustre: server umount lustre-OST0003 complete [ 9221.111279] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.122624] Lustre: Skipped 1 previous similar message [ 9221.132387] LustreError: 195298:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9221.146623] LustreError: 195298:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 15 previous similar messages [ 9236.486609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.495376] Lustre: Skipped 5 previous similar messages [ 9242.963952] Lustre: server umount lustre-MDT0000 complete [ 9247.327913] LustreError: 193284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778625144 with bad export cookie 13212964976661416806 [ 9247.332271] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.345013] LustreError: 193284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9251.813832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.846126] Lustre: Skipped 6 previous similar messages [ 9253.838510] Lustre: server umount lustre-MDT0001 complete [ 9258.138708] Lustre: server umount lustre-OST0000 complete [ 9261.979130] Lustre: server umount lustre-OST0001 complete [ 9269.670214] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9279.180151] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9326.257217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.578497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9326.667843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9326.787301] Lustre: lustre-MDT0000: new disk, initializing [ 9326.977734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.980835] Lustre: Skipped 1 previous similar message [ 9327.004426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9332.447935] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9345.308635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9345.478066] Lustre: 203087:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9345.603065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9345.612967] Lustre: Skipped 1 previous similar message [ 9345.787296] Lustre: lustre-MDT0001: new disk, initializing [ 9345.930743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9345.950796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9351.638476] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9360.258503] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9365.129422] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9373.565747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.288824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9375.489865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9381.307661] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9389.983553] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9395.310946] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9395.637444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9401.471953] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9401.779639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9406.435310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9406.435495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.457714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9406.476801] Lustre: Skipped 1 previous similar message [ 9406.499216] Lustre: Skipped 9 previous similar messages [ 9418.208149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9418.429828] Lustre: server umount lustre-OST0000 complete [ 9429.968417] Lustre: server umount lustre-MDT0000 complete [ 9432.035218] LustreError: 203096:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.068849] LustreError: 203096:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 16 previous similar messages [ 9434.330204] LustreError: 203081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778625331 with bad export cookie 13212964976661419102 [ 9434.339523] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.973510] Lustre: server umount lustre-MDT0001 complete [ 9439.236555] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:35:33 (1778625333) [ 9458.910492] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9470.565198] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9521.887180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.639938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9522.748468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9522.759997] Lustre: Skipped 1 previous similar message [ 9523.023977] Lustre: lustre-MDT0000: new disk, initializing [ 9523.039385] Lustre: Skipped 1 previous similar message [ 9523.261712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9523.275939] Lustre: Skipped 1 previous similar message [ 9529.845779] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9540.720535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9540.807759] Lustre: 209268:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9541.016088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9545.532919] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9555.058518] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9560.701911] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9569.169189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9570.988118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9577.718781] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9587.489690] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9593.203341] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9593.370963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9599.447264] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9599.734842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9606.646107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.647560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9606.664443] Lustre: Skipped 2 previous similar messages [ 9606.665355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9606.665360] Lustre: Skipped 6 previous similar messages [ 9606.715599] LustreError: Skipped 1 previous similar message [ 9608.419030] Lustre: server umount lustre-OST0000 complete [ 9619.244068] Lustre: server umount lustre-MDT0000 complete [ 9621.985082] LustreError: 209278:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.020429] LustreError: 209278:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9623.900105] LustreError: 210047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778625520 with bad export cookie 13212964976661419984 [ 9623.910566] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9623.914242] LustreError: 210047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9624.231510] Lustre: server umount lustre-MDT0001 complete [ 9635.086712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.131711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.138971] Lustre: Skipped 5 previous similar messages [ 9641.778706] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9652.385588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.139572] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9666.138982] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9671.418632] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9681.011069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9690.290082] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9697.837874] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9715.687437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9717.971187] Lustre: server umount lustre-OST0000 complete [ 9730.024021] Lustre: server umount lustre-MDT0000 complete [ 9734.978701] LustreError: 215642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778625631 with bad export cookie 13212964976661420866 [ 9734.989481] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9735.009366] LustreError: 215642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9735.055031] LustreError: 182863:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting [ 9735.592291] Lustre: server umount lustre-MDT0001 complete [ 9836.784914] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 9841.233112] Key type lgssc unregistered [ 9841.634727] LNet: 216411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9841.641154] LNetError: 216411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9841.671173] LNet: Removed LNI 192.168.201.134@tcp [ 9842.874962] Key type .llcrypt unregistered [ 9842.877836] Key type ._llcrypt unregistered [ 9878.438530] Key type ._llcrypt registered [ 9878.441340] Key type .llcrypt registered [ 9878.619795] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9900.728848] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9902.079364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9902.250323] alg: No test for adler32 (adler32-zlib) [ 9903.628812] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [ 9904.069442] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 9905.768265] Key type lgssc registered [ 9907.416578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9955.506924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9955.557227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.058881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9957.136578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9957.346622] Lustre: lustre-MDT0000: new disk, initializing [ 9957.692182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9957.744473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9964.429584] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9977.678448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9977.897717] Lustre: 219736:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9977.985988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9977.992175] Lustre: Skipped 1 previous similar message [ 9978.111678] Lustre: lustre-MDT0001: new disk, initializing [ 9978.316839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9978.406204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9978.434037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9983.711319] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9992.685365] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9997.046669] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10005.515685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.887541] Lustre: lustre-OST0000: new disk, initializing [10005.895977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10005.996684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10012.794593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10012.818589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10013.022355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10014.297226] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10023.904556] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10029.605946] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10029.963508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10035.298245] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10035.527919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10038.760963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10038.779618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10038.780573] LustreError: Skipped 1 previous similar message [10038.783610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10038.800661] Lustre: Skipped 1 previous similar message [10038.818915] Lustre: Skipped 1 previous similar message [10043.874420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10043.888137] Lustre: Skipped 1 previous similar message [10049.008551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10049.018367] Lustre: Skipped 1 previous similar message [10052.064971] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10052.427825] Lustre: server umount lustre-OST0000 complete [10059.236530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10059.241110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10059.252457] Lustre: Skipped 1 previous similar message [10059.289486] Lustre: Skipped 1 previous similar message [10063.265338] Lustre: server umount lustre-MDT0000 complete [10067.533144] LustreError: 219728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778625964 with bad export cookie 5022974814052470026 [10067.535746] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.550467] LustreError: 219728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10068.059679] Lustre: server umount lustre-MDT0001 complete [10089.399779] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10101.943723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10102.595073] LustreError: 224268:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.611950] LustreError: 224268:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10102.649166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10106.964488] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10107.881979] LustreError: 224269:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.994642] LustreError: 224268:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.997514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.381711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10123.017026] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10126.575357] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10134.202707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10134.558116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10141.545414] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10151.229027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.546282] Lustre: lustre-OST0001: new disk, initializing [10151.550342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10151.672929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10158.868324] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10160.795717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10160.814722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10160.974083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10171.831965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10182.757852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10197.999424] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:48:12 (1778626092) [10202.083403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.090696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10202.109793] Lustre: Skipped 3 previous similar messages [10202.125221] Lustre: Skipped 3 previous similar messages [10206.763855] Lustre: server umount lustre-MDT0000 complete [10207.224325] LustreError: 224268:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.251938] LustreError: 224268:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10211.433977] LustreError: 224248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778626108 with bad export cookie 5022974814052470908 [10211.435491] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10211.443876] LustreError: 224248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10211.854825] Lustre: server umount lustre-MDT0001 complete [10225.490039] Lustre: server umount lustre-OST0000 complete [10228.320236] Lustre: 217960:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626109/real 1778626109] req@ffff8c6651e1e680 x1865024326978560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778626125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10228.407992] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10230.333312] Lustre: server umount lustre-OST0001 complete [10237.947422] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [10248.038555] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10294.338464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.558863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10294.591606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10294.668579] Lustre: lustre-MDT0000: new disk, initializing [10294.765291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10294.782600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10299.529596] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10310.955027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.103832] Lustre: 230422:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10311.159424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10311.166718] Lustre: Skipped 1 previous similar message [10311.242655] Lustre: lustre-MDT0001: new disk, initializing [10311.419386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10311.436735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10317.141765] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10325.726811] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10330.188081] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10337.846047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10338.093944] Lustre: lustre-OST0000: new disk, initializing [10338.097874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10340.148255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10340.163329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10340.278472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10345.572925] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10354.193918] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10358.843896] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10359.023784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10364.787507] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10365.047282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10371.042240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.044678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10371.060963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10371.070796] Lustre: Skipped 1 previous similar message [10373.789464] Lustre: server umount lustre-OST0000 complete [10381.281710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.317019] Lustre: Skipped 2 previous similar messages [10384.900716] Lustre: server umount lustre-MDT0000 complete [10386.401932] LustreError: 232818:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.416211] LustreError: 232818:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [10389.046777] LustreError: 232163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778626285 with bad export cookie 5022974814052473043 [10389.066363] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.072344] LustreError: 232163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10389.714850] Lustre: server umount lustre-MDT0001 complete [10400.652557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.398664] LustreError: 234235:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.594533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10401.612652] Lustre: Skipped 2 previous similar messages [10406.026074] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10415.073645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.216842] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10427.670485] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10432.752953] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10441.318468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.129503] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10453.986374] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10604.002458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10604.007785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10604.016899] Lustre: Skipped 1 previous similar message [10604.028384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10604.043416] Lustre: Skipped 3 previous similar messages [10607.530241] Lustre: server umount lustre-OST0000 complete [10617.553946] Lustre: server umount lustre-MDT0000 complete [10619.361579] LustreError: 234236:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10619.377673] LustreError: 234236:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [10621.595885] LustreError: 234218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778626518 with bad export cookie 5022974814052473925 [10621.605363] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10621.617514] LustreError: 234218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10621.939223] Lustre: server umount lustre-MDT0001 complete [10722.889450] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [10726.542805] Key type lgssc unregistered [10726.855584] LNet: 237488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10726.870038] LNetError: 237488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10726.899935] LNet: Removed LNI 192.168.201.134@tcp [10727.845137] Key type .llcrypt unregistered [10727.847164] Key type ._llcrypt unregistered [10759.172288] Key type ._llcrypt registered [10759.174457] Key type .llcrypt registered [10759.277501] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [10775.558524] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10776.886566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10777.122316] alg: No test for adler32 (adler32-zlib) [10778.151238] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [10778.351641] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [10780.000216] Key type lgssc registered [10781.302987] Lustre: Echo OBD driver; http://www.lustre.org/ [10829.576496] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10841.339942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10841.366705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.663430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10842.713044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10842.805984] Lustre: lustre-MDT0000: new disk, initializing [10842.920020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10842.941607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10847.864855] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10859.155031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10859.325696] Lustre: 241510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10859.438258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10859.441847] Lustre: Skipped 1 previous similar message [10859.566950] Lustre: lustre-MDT0001: new disk, initializing [10859.725532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10859.799492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10859.812843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10864.405485] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10870.028852] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10877.655526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10877.922944] Lustre: lustre-OST0000: new disk, initializing [10877.927458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10878.041899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10883.246337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10883.275304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10883.348210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10884.930049] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10897.404313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10897.672791] Lustre: lustre-OST0001: new disk, initializing [10897.676156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10897.783873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10905.162675] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10907.759285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10907.770547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10907.852203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10915.501197] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10921.434875] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10933.733246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.738339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10933.744935] Lustre: Skipped 1 previous similar message [10933.755223] Lustre: Skipped 2 previous similar messages [10938.148652] Lustre: server umount lustre-MDT0000 complete [10938.853851] LustreError: 242562:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.864416] LustreError: 242562:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [10941.763302] LustreError: 241501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778626838 with bad export cookie 7343883567345447320 [10941.768452] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10941.770918] LustreError: 241501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10942.224177] Lustre: server umount lustre-MDT0001 complete [10956.629810] Lustre: server umount lustre-OST0000 complete [10959.138226] Lustre: 239034:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626840/real 1778626840] req@ffff8c664ce46d80 x1865025244441088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778626856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10959.166542] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10959.174552] Lustre: Skipped 2 previous similar messages [10960.668081] Lustre: server umount lustre-OST0001 complete [10977.028475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10987.003422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10997.138096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11007.844249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11030.648803] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11042.916509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.952187] Lustre: lustre-MDT0000: reset Object Index mappings [11043.468791] LustreError: 248950:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.576515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11048.201727] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11048.932306] LustreError: 248951:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.054339] LustreError: 248950:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.411395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.452295] Lustre: lustre-MDT0001: reset Object Index mappings [11058.315908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11063.571184] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11067.034242] Lustre: 250056:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11074.530597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11074.946223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11081.310712] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11083.115526] LustreError: 250409:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11089.413586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11091.039683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11096.591989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11098.943732] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11107.241297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11111.507446] Lustre: 251897:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11117.026642] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.029811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.041116] Lustre: Skipped 1 previous similar message [11119.489559] Lustre: server umount lustre-MDT0000 complete [11122.152178] LustreError: 248951:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.183415] LustreError: 248951:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [11123.652852] LustreError: 248932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778627020 with bad export cookie 7343883567345454103 [11123.684608] LustreError: 248932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11123.692261] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11124.348939] Lustre: server umount lustre-MDT0001 complete [11139.921615] Lustre: server umount lustre-OST0000 complete [11143.641186] Lustre: 239032:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778627024/real 1778627024] req@ffff8c6644ae1180 x1865025244491520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778627040 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11154.114465] Lustre: server umount lustre-OST0001 complete [11173.784152] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:04:28 (1778627068) [11190.097621] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [11200.208637] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11241.668869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.093768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11242.120996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11242.185655] Lustre: lustre-MDT0000: new disk, initializing [11242.287533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11242.304640] Lustre: Skipped 1 previous similar message [11242.332923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11247.254768] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11259.031644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.238224] Lustre: 256134:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11259.364691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11259.372610] Lustre: Skipped 1 previous similar message [11259.538339] Lustre: lustre-MDT0001: new disk, initializing [11259.726149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11259.736463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11265.148586] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11273.482936] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11277.147157] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11285.314985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.687407] Lustre: lustre-OST0000: new disk, initializing [11285.692019] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11286.994182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11287.014648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11287.189694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11294.274397] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11303.535991] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11309.437498] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11309.704542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11315.767681] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11316.066043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11318.242656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11318.247968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.257467] Lustre: Skipped 2 previous similar messages [11318.261184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11318.266731] Lustre: Skipped 3 previous similar messages [11322.855335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11322.855791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11322.856207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.886044] Lustre: Skipped 1 previous similar message [11324.586178] Lustre: server umount lustre-OST0000 complete [11331.554650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.563630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.572349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11335.432969] Lustre: server umount lustre-MDT0000 complete [11338.220262] LustreError: 256145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.242980] LustreError: 256145:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [11340.905358] LustreError: 256924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778627237 with bad export cookie 7343883567345456203 [11340.934450] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.540679] Lustre: server umount lustre-MDT0001 complete [11364.273330] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11374.894806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.328226] LustreError: 260646:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.365850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11375.368646] Lustre: Skipped 2 previous similar messages [11379.092910] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11389.909374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.112768] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11399.115624] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11406.423504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11413.038575] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11423.295536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11423.510430] Lustre: lustre-OST0001: new disk, initializing [11423.521510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11425.339481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11425.354573] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11425.520181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11430.601396] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11442.604650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11447.561471] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11453.102453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11453.184947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11461.606616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.623066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11461.630784] Lustre: Skipped 4 previous similar messages [11461.638618] Lustre: Skipped 4 previous similar messages [11467.107601] Lustre: server umount lustre-MDT0000 complete [11471.523080] LustreError: 260629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778627368 with bad export cookie 7343883567345457085 [11471.534638] LustreError: 260629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11471.561081] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11471.844923] LustreError: 262118:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.861240] LustreError: 262118:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [11471.973596] Lustre: server umount lustre-MDT0001 complete [11478.458085] Lustre: server umount lustre-OST0000 complete [11482.429566] Lustre: server umount lustre-OST0001 complete [11504.762963] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11520.467528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11521.078287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11521.085747] Lustre: Skipped 3 previous similar messages [11526.065475] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11535.348261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.080545] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11543.689747] Lustre: 267036:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11552.280116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11559.642942] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11566.045533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11568.938338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11574.350655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11577.037679] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11586.816354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11591.401666] Lustre: 268880:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11597.988920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11598.034844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11609.777455] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:11:43 (1778627503) [11625.444091] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.461153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.465774] Lustre: Skipped 4 previous similar messages [11625.490492] Lustre: Skipped 7 previous similar messages [11627.475678] Lustre: server umount lustre-MDT0000 complete [11630.565334] LustreError: 265928:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.577753] LustreError: 265928:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 13 previous similar messages [11631.708435] LustreError: 265910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778627528 with bad export cookie 7343883567345463854 [11631.709306] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11631.717928] LustreError: 265910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11638.241952] Lustre: server umount lustre-MDT0001 complete [11648.780492] Lustre: server umount lustre-OST0000 complete [11652.959811] Lustre: server umount lustre-OST0001 complete [11659.490482] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [11668.638469] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11722.730479] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11731.845506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11732.114256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11732.147806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11732.241803] Lustre: lustre-MDT0000: new disk, initializing [11732.356666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11737.287884] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11749.339533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11749.450534] Lustre: 273069:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11749.666628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11754.733840] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11767.198266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.777723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11773.222825] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11784.369693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.776940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11784.780896] Lustre: Skipped 6 previous similar messages [11786.360350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11786.368643] Lustre: Skipped 1 previous similar message [11786.477524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11791.686626] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11801.369985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11804.951855] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11804.956356] Lustre: Skipped 1 previous similar message [11820.034702] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:15:14 (1778627714) [11822.597152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.603777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.619099] Lustre: Skipped 6 previous similar messages [11822.642664] Lustre: Skipped 7 previous similar messages [11824.856157] Lustre: server umount lustre-MDT0000 complete [11828.784876] LustreError: 273856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778627725 with bad export cookie 7343883567345471309 [11828.798747] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11828.815278] LustreError: 273856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11829.097080] Lustre: server umount lustre-MDT0001 complete [11842.211971] Lustre: server umount lustre-OST0000 complete [11857.066968] Lustre: server umount lustre-OST0001 complete [11866.213994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11872.702855] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11883.202760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.044193] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11896.689840] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11902.199505] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11910.474938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11912.066505] LustreError: 279358:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11912.093993] LustreError: 279358:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 13 previous similar messages [11918.049804] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11925.221603] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11946.454218] loop1: detected capacity change from 0 to 204800000 [11946.747834] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.776321] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.787899] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11952.613382] loop1: detected capacity change from 0 to 204800000 [11959.299944] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11959.417853] LustreError: 277740:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs24106: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11959.428586] LustreError: 277740:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs24106 [11959.431839] LustreError: 277740:0:(mgs_handler.c:517:mgs_target_reg()) fs24106-MDT0001: Error checking target index: rc = -13 [11959.450145] LustreError: fs24106-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11959.461918] LustreError: 281179:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11959.467871] LustreError: 281179:0:(tgt_mount.c:2084:server_put_super()) no obd fs24106-MDT0001 [11959.476263] LustreError: 281179:0:(tgt_mount.c:116:server_deregister_mount()) fs24106-MDT0001 not registered [11959.546581] Lustre: server umount fs24106-MDT0001 complete [11959.552803] LustreError: 281179:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11970.031722] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.100677] Lustre: Setting parameter fs24106-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24106-MDT0001 [11970.133845] Lustre: srv-fs24106-MDT0001: No data found on store. Initialize space. [11970.140701] Lustre: Skipped 4 previous similar messages [11970.214633] Lustre: fs24106-MDT0001: new disk, initializing [11970.219143] Lustre: Skipped 3 previous similar messages [11970.355119] Lustre: srv-fs24106-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11970.367158] Lustre: 281647:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24106-MDT0001-mdtlov: can't create new llog: rc = -115 [11970.378255] LustreError: 281647:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs24106-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11975.119945] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11984.094273] Lustre: server umount fs24106-MDT0001 complete [11989.995854] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12003.547593] loop1: detected capacity change from 0 to 204800000 [12010.468551] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12010.919520] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.330569] Lustre: 283199:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24106-MDT0001-mdtlov: can't create new llog: rc = -115 [12011.341183] LustreError: 283199:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs24106-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12017.243549] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12022.524684] Lustre: server umount fs24106-MDT0001 complete [12036.070907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.080805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12036.097265] Lustre: Skipped 1 previous similar message [12036.115285] Lustre: Skipped 2 previous similar messages [12040.235337] Lustre: server umount lustre-MDT0000 complete [12050.105902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12050.256469] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12055.765596] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12056.048723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12064.505371] loop1: detected capacity change from 0 to 204800000 [12070.962847] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.366523] Lustre: 285616:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24106-MDT0001-mdtlov: can't create new llog: rc = -115 [12071.375129] LustreError: 285616:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs24106-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12075.502223] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12081.070810] Lustre: server umount fs24106-MDT0001 complete [12112.405512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12116.285422] Lustre: server umount lustre-OST0000 complete [12122.596141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12134.368268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12134.513358] Lustre: server umount lustre-MDT0000 complete [12138.344635] LustreError: 278476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778628035 with bad export cookie 7343883567345473990 [12138.345414] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12138.376326] LustreError: 278476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12138.752075] Lustre: server umount lustre-MDT0001 complete [12152.212431] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12155.181474] Key type lgssc unregistered [12155.485721] LNet: 287927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12155.490103] LNetError: 287927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12155.505261] LNet: Removed LNI 192.168.201.134@tcp [12156.453544] Key type .llcrypt unregistered [12156.458988] Key type ._llcrypt unregistered [12176.430359] Key type ._llcrypt registered [12176.432592] Key type .llcrypt registered [12176.502940] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [12177.875337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12177.937968] alg: No test for adler32 (adler32-zlib) [12179.026409] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [12179.237440] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [12180.888210] Key type lgssc registered [12182.511045] Lustre: Echo OBD driver; http://www.lustre.org/ [12192.128672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12192.159947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.719045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12197.693841] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12204.001878] LustreError: 289185:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.014359] LustreError: 289185:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [12205.889845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.206552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12210.957216] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12216.295865] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12219.755444] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12227.007536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12227.711627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12231.817536] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.581401] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12237.283695] LustreError: 290789:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12243.443648] LustreError: 290789:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12243.468104] LustreError: 290789:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [12243.732589] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12248.550545] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.574361] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [12258.809514] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.827561] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [12266.000924] loop1: detected capacity change from 0 to 204800000 [12266.082243] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.097162] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.105783] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12272.064804] loop1: detected capacity change from 0 to 204800000 [12276.501586] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.524415] LustreError: 290786:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [12278.225399] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12278.348619] Lustre: fs10992-OST0000: new disk, initializing [12278.352473] Lustre: srv-fs10992-OST0000: No data found on store. Initialize space. [12278.422332] Lustre: fs10992-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12286.468633] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12297.119188] Lustre: server umount fs10992-OST0000 complete [12310.001038] LustreError: 290788:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.037381] LustreError: 290788:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 19 previous similar messages [12311.609478] loop1: detected capacity change from 0 to 204800000 [12315.107233] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12317.554398] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12317.851247] Lustre: fs10992-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12320.234254] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12324.396260] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12335.183748] loop2: detected capacity change from 0 to 204800000 [12335.466492] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12335.487918] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12335.508730] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12341.631913] loop2: detected capacity change from 0 to 204800000 [12347.562486] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12347.658301] LustreError: fs10992-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12347.683826] LustreError: fs10992-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12347.691439] LustreError: 295638:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [12347.699687] LustreError: 295638:0:(tgt_mount.c:2084:server_put_super()) no obd fs10992-OST0001 [12347.704512] LustreError: 295638:0:(tgt_mount.c:116:server_deregister_mount()) fs10992-OST0001 not registered [12347.737970] Lustre: server umount fs10992-OST0001 complete [12347.740751] LustreError: 295638:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12359.335425] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12359.675842] Lustre: fs10992-OST0001: new disk, initializing [12359.684637] Lustre: srv-fs10992-OST0001: No data found on store. Initialize space. [12359.789613] Lustre: fs10992-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12367.411783] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12376.555313] LustreError: 290788:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12376.607308] LustreError: 290788:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 37 previous similar messages [12376.842751] Lustre: server umount fs10992-OST0000 complete [12387.086911] Lustre: server umount fs10992-OST0001 complete [12400.616252] loop1: detected capacity change from 0 to 204800000 [12406.849544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12407.173988] Lustre: fs10992-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12413.630412] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12421.076977] loop2: detected capacity change from 0 to 204800000 [12427.228649] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12433.859186] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12443.351806] Lustre: server umount fs10992-OST0000 complete [12452.937696] Lustre: server umount fs10992-OST0001 complete [12467.681768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12467.694059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.702338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12468.705889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.711348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12468.744103] Lustre: Skipped 1 previous similar message [12473.132437] Lustre: server umount lustre-OST0000 complete [12477.411554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12477.432786] LustreError: Skipped 2 previous similar messages [12477.438842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.453670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12483.400501] Lustre: server umount lustre-MDT0000 complete [12487.294824] LustreError: 296188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778628384 with bad export cookie 5925090209623124097 [12487.296614] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12487.305993] LustreError: 296188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12487.694820] Lustre: server umount lustre-MDT0001 complete [12496.448691] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12499.061548] Key type lgssc unregistered [12499.319008] LNet: 302044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12499.346256] LNetError: 302044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12499.376980] LNet: Removed LNI 192.168.201.134@tcp [12500.138236] Key type .llcrypt unregistered [12500.144156] Key type ._llcrypt unregistered [12533.814443] Key type ._llcrypt registered [12533.817484] Key type .llcrypt registered [12533.918787] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [12548.454437] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [12549.562328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12549.616654] alg: No test for adler32 (adler32-zlib) [12550.641484] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [12550.898433] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [12552.664268] Key type lgssc registered [12553.636334] Lustre: Echo OBD driver; http://www.lustre.org/ [12590.569214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12590.631319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12592.003150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12592.057892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12592.227525] Lustre: lustre-MDT0000: new disk, initializing [12592.309044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12592.329719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12595.833354] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12607.341749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12607.530908] Lustre: 305873:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12607.614292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12607.623901] Lustre: Skipped 1 previous similar message [12607.739186] Lustre: lustre-MDT0001: new disk, initializing [12607.870486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12607.914348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12607.931186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12613.010420] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12621.563590] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12624.941199] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12632.556576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12632.986227] Lustre: lustre-OST0000: new disk, initializing [12632.992777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12633.084295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12638.831028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12638.849152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12639.001638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12639.430422] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12647.071252] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12649.254979] Lustre: Mounted lustre-client [12650.248865] LustreError: 308368:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c6651b28000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [12650.288704] Lustre: Unmounted lustre-client [12659.683494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.687990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12659.691152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12659.700592] Lustre: Skipped 1 previous similar message [12659.721585] Lustre: Skipped 1 previous similar message [12664.822586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12664.839246] Lustre: Skipped 1 previous similar message [12665.582317] Lustre: server umount lustre-OST0000 complete [12669.921957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12669.944950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12674.531761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12674.537896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.574760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12675.820432] Lustre: server umount lustre-MDT0000 complete [12679.350530] LustreError: 305865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778628576 with bad export cookie 2275700179511376853 [12679.360353] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12679.363827] LustreError: 305865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12679.602105] Lustre: server umount lustre-MDT0001 complete [12689.477428] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12692.113483] Key type lgssc unregistered [12706.592192] LNet: 1 local NIs in recovery (showing 1): 192.168.201.134@tcp [12706.723677] LNet: 309647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12706.731486] LNetError: 309647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12781.488522] LNet: Removed LNI 192.168.201.134@tcp [12782.553751] Key type .llcrypt unregistered [12782.557474] Key type ._llcrypt unregistered [12806.571362] Key type ._llcrypt registered [12806.575516] Key type .llcrypt registered [12806.761180] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [12824.566899] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [12826.204164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12826.401047] alg: No test for adler32 (adler32-zlib) [12827.510098] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [12827.786636] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [12829.456988] Key type lgssc registered [12830.420335] Lustre: Echo OBD driver; http://www.lustre.org/ [12866.033880] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:32:40 (1778628760) [12873.879247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12873.922651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12875.251796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12875.308596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12875.438775] Lustre: lustre-MDT0000: new disk, initializing [12875.577538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12875.603891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12880.915531] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12891.842959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12892.043466] Lustre: 312850:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12892.107326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12892.112674] Lustre: Skipped 1 previous similar message [12892.186895] Lustre: lustre-MDT0001: new disk, initializing [12892.297938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12892.350708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12892.361264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12897.688795] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12906.664958] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12911.977557] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12920.036642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12920.422101] Lustre: lustre-OST0000: new disk, initializing [12920.430908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12920.549875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12928.257049] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12929.063641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12929.071577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12929.193143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12937.830062] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12949.478061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12949.478859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12949.537358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12954.595262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12954.617196] Lustre: Skipped 1 previous similar message [12955.124119] Lustre: server umount lustre-OST0000 complete [12959.714103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12959.716111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12959.732457] Lustre: Skipped 2 previous similar messages [12959.741313] Lustre: Skipped 2 previous similar messages [12964.842800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.854504] Lustre: Skipped 1 previous similar message [12965.293831] Lustre: server umount lustre-MDT0000 complete [12969.242493] LustreError: 312843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778628866 with bad export cookie 16159544775779421413 [12969.256451] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12969.258317] LustreError: 312843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12969.825390] Lustre: server umount lustre-MDT0001 complete [12981.551440] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12984.623209] Key type lgssc unregistered [12984.970223] LNet: 316222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12984.978918] LNetError: 316222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12986.023779] LNet: Removed LNI 192.168.201.134@tcp [12986.988705] Key type .llcrypt unregistered [12986.992095] Key type ._llcrypt unregistered [13018.389691] Key type ._llcrypt registered [13018.395092] Key type .llcrypt registered [13018.678906] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [13021.113740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13021.132068] alg: No test for adler32 (adler32-zlib) [13022.374686] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [13022.783487] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [13024.584780] Key type lgssc registered [13026.472290] Lustre: Echo OBD driver; http://www.lustre.org/ [13035.350643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13035.396640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13036.886586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13042.810911] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13046.250645] LustreError: 317630:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.264229] LustreError: 317630:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [13051.393970] LustreError: 317629:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13053.500558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13054.480881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13059.702316] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13065.132484] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13069.430967] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13077.738358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13078.152324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13085.911905] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13092.402896] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13101.941370] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13103.896276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13110.566175] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [13110.753568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13110.762309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.291132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.295359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13112.336221] Lustre: Skipped 1 previous similar message [13112.338872] Lustre: Skipped 2 previous similar messages [13114.607501] Lustre: server umount lustre-MDT0000 complete [13115.662460] LustreError: 317626:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13119.294774] LustreError: 317611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778629016 with bad export cookie 12633199193433032340 [13119.312591] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13119.318079] LustreError: 317611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13120.784407] LustreError: 317625:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.804745] LustreError: 317625:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [13120.815762] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [13120.815771] Lustre: Skipped 1 previous similar message [13122.530135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13124.583321] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13124.600992] Lustre: Skipped 1 previous similar message [13125.829310] Lustre: server umount lustre-MDT0001 complete [13130.135155] Lustre: server umount lustre-OST0000 complete [13140.153505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.212791] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13147.919827] Lustre: 321357:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.134@tcp [13150.514267] Lustre: server umount MGS complete [13159.343695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13160.053517] LustreError: 321929:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13160.066641] LustreError: 321929:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [13160.158693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13164.942763] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13170.660458] LustreError: 321930:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13170.690329] LustreError: 321930:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [13173.494213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.883499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.446869] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13187.228861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13187.568122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13194.355745] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13201.349668] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13203.297479] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13207.238389] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13213.313308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [13219.298179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13219.310957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13219.346797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13220.519265] Lustre: server umount lustre-OST0000 complete [13226.466187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13231.585262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13231.602202] Lustre: Skipped 3 previous similar messages [13235.775640] Lustre: server umount lustre-MDT0000 complete [13236.709365] LustreError: 321926:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.741891] LustreError: 321926:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [13239.983328] LustreError: 321910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778629136 with bad export cookie 12633199193433033180 [13239.993405] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13240.010352] LustreError: 321910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13240.565407] Lustre: server umount lustre-MDT0001 complete [13253.286565] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13256.162942] Key type lgssc unregistered [13256.400375] LNet: 325105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13256.411203] LNetError: 325105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13257.451045] LNet: Removed LNI 192.168.201.134@tcp [13258.453132] Key type .llcrypt unregistered [13258.454834] Key type ._llcrypt unregistered [13283.190626] Key type ._llcrypt registered [13283.192284] Key type .llcrypt registered [13283.386808] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [13304.345609] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [13305.848947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13306.182982] alg: No test for adler32 (adler32-zlib) [13307.516631] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [13307.927445] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [13309.696184] Key type lgssc registered [13311.355838] Lustre: Echo OBD driver; http://www.lustre.org/ [13355.390692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13355.445494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13356.828603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13356.885729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13356.968875] Lustre: lustre-MDT0000: new disk, initializing [13357.076166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13357.091774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13362.822049] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13374.405423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13374.630465] Lustre: 328239:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13374.726362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13374.736959] Lustre: Skipped 1 previous similar message [13374.869983] Lustre: lustre-MDT0001: new disk, initializing [13374.996889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13375.077067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13375.101690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13381.389913] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13391.179300] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13395.691445] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13404.379881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13404.639825] Lustre: lustre-OST0000: new disk, initializing [13404.643316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13404.723835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13411.403868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13411.411631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13411.546786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13411.831419] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13420.326759] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13425.983734] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13426.370738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13431.855507] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13432.226079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13437.409148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13437.415670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13437.422822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.434225] Lustre: Skipped 1 previous similar message [13437.445980] Lustre: Skipped 1 previous similar message [13442.542453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13442.560980] Lustre: Skipped 1 previous similar message [13447.657470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13447.662824] Lustre: Skipped 1 previous similar message [13449.184115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13449.424148] Lustre: server umount lustre-OST0000 complete [13456.867711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13456.875592] LustreError: Skipped 1 previous similar message [13456.887696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13456.912299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13457.896697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13460.302158] Lustre: server umount lustre-MDT0000 complete [13464.534937] LustreError: 328229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778629361 with bad export cookie 13947048473061449900 [13464.539824] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13464.546195] LustreError: 328229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13464.975339] Lustre: server umount lustre-MDT0001 complete [13475.501459] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:42:49 (1778629369) [13493.793409] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [13504.760659] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [13547.775038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13548.162119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13548.198844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13548.324611] Lustre: lustre-MDT0000: new disk, initializing [13548.521985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13548.542497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13552.947120] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13566.410197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.520868] Lustre: 334563:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13566.574104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13566.578485] Lustre: Skipped 1 previous similar message [13566.665112] Lustre: lustre-MDT0001: new disk, initializing [13566.807207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13566.851771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13566.860723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13572.011497] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13581.268257] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13587.773398] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13597.739236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13598.214190] Lustre: lustre-OST0000: new disk, initializing [13598.218815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13598.335914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13606.032901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13606.044528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13606.220745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13607.299990] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [13616.254788] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13662.693180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13662.696747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13662.705295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13662.705304] Lustre: Skipped 2 previous similar messages [13662.715105] LustreError: Skipped 1 previous similar message [13662.760592] Lustre: Skipped 1 previous similar message [13667.248062] Lustre: server umount lustre-OST0000 complete [13672.932414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13672.953303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13672.963322] Lustre: Skipped 1 previous similar message [13674.312171] Lustre: server umount lustre-MDT0000 complete [13678.049990] LustreError: 335738:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.062898] LustreError: 335738:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [13679.933698] LustreError: 334555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778629576 with bad export cookie 13947048473061450782 [13679.941547] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13679.945984] LustreError: 334555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13680.355992] Lustre: server umount lustre-MDT0001 complete [13693.637914] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13696.739659] Key type lgssc unregistered [13697.017971] LNet: 338358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13697.025857] LNetError: 338358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13697.043848] LNet: Removed LNI 192.168.201.134@tcp [13697.690528] Key type .llcrypt unregistered [13697.692409] Key type ._llcrypt unregistered [13808.941492] Key type ._llcrypt registered [13808.943804] Key type .llcrypt registered [13809.064545] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13809.367364] Key type .llcrypt unregistered [13809.369845] Key type ._llcrypt unregistered [13892.429588] Key type ._llcrypt registered [13892.432774] Key type .llcrypt registered [13892.577972] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13892.946566] Key type .llcrypt unregistered [13892.949790] Key type ._llcrypt unregistered [13924.917632] Key type ._llcrypt registered [13924.922774] Key type .llcrypt registered [13925.046654] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13925.437355] Key type .llcrypt unregistered [13925.440672] Key type ._llcrypt unregistered [13956.322370] Key type ._llcrypt registered [13956.324901] Key type .llcrypt registered [13956.455181] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13956.696358] Key type .llcrypt unregistered [13956.697969] Key type ._llcrypt unregistered [14032.621356] Key type ._llcrypt registered [14032.623908] Key type .llcrypt registered [14032.750916] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14033.075490] Key type .llcrypt unregistered [14033.078299] Key type ._llcrypt unregistered [14065.345978] Key type ._llcrypt registered [14065.348321] Key type .llcrypt registered [14065.470897] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14065.787341] Key type .llcrypt unregistered [14065.790922] Key type ._llcrypt unregistered [14094.076690] Key type ._llcrypt registered [14094.078651] Key type .llcrypt registered [14094.160343] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14094.390410] Key type .llcrypt unregistered [14094.391752] Key type ._llcrypt unregistered [14123.765618] Key type ._llcrypt registered [14123.768402] Key type .llcrypt registered [14123.897075] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14124.292673] Key type .llcrypt unregistered [14124.296077] Key type ._llcrypt unregistered [14153.543628] Key type ._llcrypt registered [14153.546129] Key type .llcrypt registered [14153.643699] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14153.865341] Key type .llcrypt unregistered [14153.867674] Key type ._llcrypt unregistered [14179.206492] Key type ._llcrypt registered [14179.209559] Key type .llcrypt registered [14179.294331] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14179.510931] Key type .llcrypt unregistered [14179.515698] Key type ._llcrypt unregistered [14235.287679] Key type ._llcrypt registered [14235.289708] Key type .llcrypt registered [14235.387534] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14235.606746] Key type .llcrypt unregistered [14235.612793] Key type ._llcrypt unregistered [14262.941855] Key type ._llcrypt registered [14262.946769] Key type .llcrypt registered [14263.053834] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14263.299583] Key type .llcrypt unregistered [14263.304228] Key type ._llcrypt unregistered [14289.714483] Key type ._llcrypt registered [14289.716827] Key type .llcrypt registered [14289.821782] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14290.042265] Key type .llcrypt unregistered [14290.043863] Key type ._llcrypt unregistered [14335.552967] Key type ._llcrypt registered [14335.554288] Key type .llcrypt registered [14335.635696] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [14348.964768] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [14350.209325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14350.232604] alg: No test for adler32 (adler32-zlib) [14351.237229] Lustre: Lustre: Build Version: 2.17.52_126_gcf32fc9 [14351.439425] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [14353.064172] Key type lgssc registered [14354.071308] Lustre: Echo OBD driver; http://www.lustre.org/ [14385.154389] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14385.177824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14386.444084] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14386.473703] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14386.543217] Lustre: tst250fs-MDT0000: new disk, initializing [14386.616457] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14386.631691] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14390.182401] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [14399.702721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14399.826643] Lustre: 358348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14399.878054] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14399.882379] Lustre: Skipped 1 previous similar message [14399.939138] Lustre: tst250fs-MDT0001: new disk, initializing [14400.046660] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14400.100187] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14400.112186] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14403.901628] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [14408.098993] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14414.233562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14414.526960] Lustre: tst250fs-OST0000: new disk, initializing [14414.534934] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14414.647808] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14420.251290] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [14423.635166] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14423.652293] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14423.793561] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14429.500735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14429.620716] Lustre: tst250fs-OST0001: new disk, initializing [14429.624203] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14429.700051] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14430.765721] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14430.773114] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14430.866754] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14434.909333] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [14440.133537] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14442.394811] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14442.403956] Lustre: Skipped 2 previous similar messages [14461.412193] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14461.413889] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14461.424978] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14461.435287] Lustre: Skipped 1 previous similar message [14461.442956] Lustre: Skipped 3 previous similar messages [14465.184262] Lustre: server umount tst250fs-MDT0000 complete [14468.032520] LustreError: 358341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778630364 with bad export cookie 9563013705043484347 [14468.043475] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14468.266186] Lustre: server umount tst250fs-MDT0001 complete [14481.968595] Lustre: server umount tst250fs-OST0000 complete [14494.292944] Lustre: server umount tst250fs-OST0001 complete [14508.176629] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [14515.442168] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [14539.777446] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:00:34 (1778630434) [14541.051877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14553.805347] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [14562.089757] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [14586.027427] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14286 sec ========= 20:01:20 (1778630480) [14587.335788] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:01:22 (1778630482) === [14590.000718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:01:24 (1778630484) === [14619.035812] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [14621.108875] Key type lgssc unregistered [14621.265787] LNet: 369829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14621.270993] LNetError: 369829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14621.284942] LNet: Removed LNI 192.168.201.134@tcp [14621.965155] Key type .llcrypt unregistered [14621.966894] Key type ._llcrypt unregistered