[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494803217 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002356] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007457] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011079] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014095] *** VALIDATE selinux *** [ 0.022354] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027050] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030134] *** VALIDATE tmpfs *** [ 0.031462] *** VALIDATE proc *** [ 0.033111] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.036213] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.044057] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.046257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047715] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053011] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056395] rcu: Hierarchical SRCU implementation. [ 0.059078] smp: Bringing up secondary CPUs ... [ 0.061099] x86: Booting SMP configuration: [ 0.062019] .... node #0, CPUs: #1 #2 #3 [ 0.069802] smp: Brought up 1 node, 4 CPUs [ 0.071018] smpboot: Max logical packages: 1 [ 0.072036] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101265] node 0 deferred pages initialised in 26ms [ 0.105129] devtmpfs: initialized [ 0.106222] x86/mm: Memory block size: 128MB [ 0.108832] gcov: version magic: 0x41383552 [ 0.110173] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.112308] pinctrl core: initialized pinctrl subsystem [ 0.113257] [ 0.113938] ************************************************************* [ 0.114012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115025] ** ** [ 0.116013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.117012] ** ** [ 0.118012] ** This means that this kernel is built to expose internal ** [ 0.119011] ** IOMMU data structures, which may compromise security on ** [ 0.120012] ** your system. ** [ 0.121011] ** ** [ 0.122012] ** If you see this message and you are not debugging the ** [ 0.123013] ** kernel, report this immediately to your vendor! ** [ 0.124013] ** ** [ 0.125010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126011] ************************************************************* [ 0.127723] NET: Registered protocol family 16 [ 0.128611] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.130083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.131606] cpuidle: using governor menu [ 0.132700] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.135346] PCI: Using configuration type 1 for base access [ 0.137106] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.145087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.146011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.148049] cryptd: max_cpu_qlen set to 1000 [ 0.150043] ACPI: Added _OSI(Module Device) [ 0.151013] ACPI: Added _OSI(Processor Device) [ 0.152011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.153012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.157485] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.160538] ACPI: Interpreter enabled [ 0.161055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.162010] ACPI: Using IOAPIC for interrupt routing [ 0.163202] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.164570] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.175256] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.176045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.177020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.178083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.180371] acpiphp: Slot [2] registered [ 0.181093] acpiphp: Slot [5] registered [ 0.182068] acpiphp: Slot [6] registered [ 0.183094] acpiphp: Slot [7] registered [ 0.184085] acpiphp: Slot [8] registered [ 0.185066] acpiphp: Slot [9] registered [ 0.185905] acpiphp: Slot [10] registered [ 0.186139] acpiphp: Slot [3] registered [ 0.187071] acpiphp: Slot [4] registered [ 0.188085] acpiphp: Slot [11] registered [ 0.189076] acpiphp: Slot [12] registered [ 0.190093] acpiphp: Slot [13] registered [ 0.191102] acpiphp: Slot [14] registered [ 0.192075] acpiphp: Slot [15] registered [ 0.193103] acpiphp: Slot [16] registered [ 0.194089] acpiphp: Slot [17] registered [ 0.195163] acpiphp: Slot [18] registered [ 0.196086] acpiphp: Slot [19] registered [ 0.197082] acpiphp: Slot [20] registered [ 0.198112] acpiphp: Slot [21] registered [ 0.199104] acpiphp: Slot [22] registered [ 0.200079] acpiphp: Slot [23] registered [ 0.201088] acpiphp: Slot [24] registered [ 0.202082] acpiphp: Slot [25] registered [ 0.203096] acpiphp: Slot [26] registered [ 0.204061] acpiphp: Slot [27] registered [ 0.205054] acpiphp: Slot [28] registered [ 0.206092] acpiphp: Slot [29] registered [ 0.207076] acpiphp: Slot [30] registered [ 0.208084] acpiphp: Slot [31] registered [ 0.209211] PCI host bridge to bus 0000:00 [ 0.211032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.213030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.217162] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.221039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.225038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.229041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.232316] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.239146] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.243617] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.264015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.267016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.271020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274406] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.279066] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.281032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.284912] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.291017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.308017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.317014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.322455] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.331017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.341027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365089] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.380465] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.399017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.420016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.444422] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.456016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.465020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.486108] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502548] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516043] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.529035] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612822] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.650020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.664868] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.678020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.712763] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.715292] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717213] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719358] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.721222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.727148] iommu: Default domain type: Passthrough [ 0.728326] SCSI subsystem initialized [ 0.729101] ACPI: bus type USB registered [ 0.730028] usbcore: registered new interface driver usbfs [ 0.731068] usbcore: registered new interface driver hub [ 0.732065] usbcore: registered new device driver usb [ 0.734182] pps_core: LinuxPPS API ver. 1 registered [ 0.736014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739072] PTP clock support registered [ 0.741039] EDAC MC: Ver: 3.0.0 [ 0.742385] PCI: Using ACPI for IRQ routing [ 0.743823] NetLabel: Initializing [ 0.744010] NetLabel: domain hash size = 128 [ 0.745009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.746128] NetLabel: unlabeled traffic allowed by default [ 0.747344] vgaarb: loaded [ 0.749100] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757534] clocksource: Switched to clocksource kvm-clock [ 0.873172] VFS: Disk quotas dquot_6.6.0 [ 0.874076] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875613] *** VALIDATE ramfs *** [ 0.876481] *** VALIDATE hugetlbfs *** [ 0.877672] pnp: PnP ACPI init [ 0.879631] pnp: PnP ACPI: found 6 devices [ 0.893067] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896032] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.897995] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900059] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.902243] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.904459] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.906945] NET: Registered protocol family 2 [ 0.909240] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913650] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917263] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.922077] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.925361] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928123] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.930806] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.932977] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936745] NET: Registered protocol family 1 [ 0.941292] RPC: Registered named UNIX socket transport module. [ 0.943299] RPC: Registered udp transport module. [ 0.944828] RPC: Registered tcp transport module. [ 0.945992] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.948340] NET: Registered protocol family 44 [ 0.951539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.956739] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960935] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.963600] PCI: CLS 0 bytes, default 64 [ 0.965155] Unpacking initramfs... [ 2.441990] debug: unmapping init [mem 0xffff8f147cc54000-0xffff8f147ffbffff] [ 2.445811] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.448019] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.450840] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.925497] Initialise system trusted keyrings [ 2.927211] Key type blacklist registered [ 2.929101] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936881] zbud: loaded [ 2.939770] *** VALIDATE nfs *** [ 2.940935] *** VALIDATE nfs4 *** [ 2.942142] pstore: using deflate compression [ 2.945304] Platform Keyring initialized [ 3.052790] NET: Registered protocol family 38 [ 3.054083] Key type asymmetric registered [ 3.055429] Asymmetric key parser 'x509' registered [ 3.057509] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.061344] io scheduler mq-deadline registered [ 3.062936] io scheduler kyber registered [ 3.064823] io scheduler bfq registered [ 3.066987] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.070562] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.073779] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.077206] ACPI: Power Button [PWRF] [ 3.082269] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.088584] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.105307] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.113251] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.128381] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.157148] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.184843] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.190290] Non-volatile memory driver v1.3 [ 3.191952] Linux agpgart interface v0.103 [ 3.223563] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.226261] vda: detected capacity change from 0 to 76681216 [ 3.271856] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.276864] vdb: detected capacity change from 0 to 1073741824 [ 3.299605] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.302861] vdc: detected capacity change from 0 to 2621440000 [ 3.347255] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.351148] vdd: detected capacity change from 0 to 2621440000 [ 3.374730] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.382589] vde: detected capacity change from 0 to 4294967296 [ 3.403569] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.406846] vdf: detected capacity change from 0 to 4294967296 [ 3.423792] libphy: Fixed MDIO Bus: probed [ 3.432133] usbcore: registered new interface driver usbserial_generic [ 3.434926] usbserial: USB Serial support registered for generic [ 3.437639] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.442717] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.444885] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.447735] mousedev: PS/2 mouse device common for all mice [ 3.451740] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.455301] rtc_cmos 00:05: RTC can wake from S4 [ 3.456265] rtc_cmos 00:05: registered as rtc0 [ 3.456332] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.456365] intel_pstate: CPU model not supported [ 3.462540] hid: raw HID events driver (C) Jiri Kosina [ 3.473229] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.475347] usbcore: registered new interface driver usbhid [ 3.475352] usbhid: USB HID core driver [ 3.476545] drop_monitor: Initializing network drop monitor service [ 3.483120] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.488680] Initializing XFRM netlink socket [ 3.489283] NET: Registered protocol family 10 [ 3.494704] Segment Routing with IPv6 [ 3.508189] NET: Registered protocol family 17 [ 3.510372] mpls_gso: MPLS GSO support [ 3.516657] RAS: Correctable Errors collector initialized. [ 3.522550] AVX version of gcm_enc/dec engaged. [ 3.524616] AES CTR mode by8 optimization enabled [ 3.604967] sched_clock: Marking stable (3604947927, 0)->(4578134387, -973186460) [ 3.608509] registered taskstats version 1 [ 3.610261] Loading compiled-in X.509 certificates [ 3.612484] zswap: loaded using pool lzo/zbud [ 3.635641] Key type big_key registered [ 3.647440] Key type encrypted registered [ 3.648809] ima: No TPM chip found, activating TPM-bypass! [ 3.650658] ima: Allocated hash algorithm: sha1 [ 3.652252] ima: No architecture policies found [ 3.653886] evm: Initialising EVM extended attributes: [ 3.655857] evm: security.selinux [ 3.657015] evm: security.ima [ 3.657992] evm: security.capability [ 3.659620] evm: HMAC attrs: 0x1 [ 3.662355] rtc_cmos 00:05: setting system clock to 2026-09-04 11:53:48 UTC (1788522828) [ 3.668273] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.671684] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.681118] Write protecting the kernel read-only data: 28672k [ 3.684285] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.687225] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.724311] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.732791] systemd[1]: Detected virtualization kvm. [ 3.734766] systemd[1]: Detected architecture x86-64. [ 3.736535] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.853780] systemd[1]: No hostname configured. [ 3.856231] systemd[1]: Set hostname to . [ 3.858507] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861714] systemd[1]: Initializing machine ID from random generator. [ 3.951158] random: ln: uninitialized urandom read (6 bytes read) [ 4.343372] random: systemd: uninitialized urandom read (16 bytes read) [ 4.346510] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.509686] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.520174] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.848735] device-mapper: uevent: version 1.0.3 [ 5.849985] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.817245] virtio_net virtio0 ens2: renamed from eth0 [ 6.835041] random: fast init done [ 7.182573] scsi host0: ata_piix [ 7.217200] scsi host1: ata_piix [ 7.222543] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.231313] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.172641] random: crng init done [ 12.174388] random: 7 urandom warning(s) missed due to ratelimiting [ 14.685667] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.612760] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.168427] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.552153] SELinux: Disabled at runtime. [ 18.615934] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.625155] systemd[1]: Detected virtualization kvm. [ 18.627195] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.409931] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.413961] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.422415] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.427208] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.432268] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.441767] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.452256] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ 20.066597] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.640179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.459187] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.470439] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.771344] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.926101] EDAC sbridge: Ver: 1.1.2 [ 25.379720] Key type dns_resolver registered [ 25.928447] NFS: Registering the id_resolver key type [ 25.931373] Key type id_resolver registered [ 25.933993] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 44.559152] hrtimer: interrupt took 4143917 ns [ 94.117480] libcfs: loading out-of-tree module taints kernel. [ 94.194597] Key type ._llcrypt registered [ 94.196232] Key type .llcrypt registered [ 94.372454] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 115.960763] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 117.740862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.757973] alg: No test for adler32 (adler32-zlib) [ 119.181130] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 120.320436] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 122.281790] Key type lgssc registered [ 124.214977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.834263] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.092295] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 203.038168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 203.105154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.459160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.494789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.582746] Lustre: lustre-MDT0000: new disk, initializing [ 204.714935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.738923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.287316] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.807449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.901739] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.929985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.933462] Lustre: Skipped 1 previous similar message [ 224.030325] Lustre: lustre-MDT0001: new disk, initializing [ 224.073456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.096426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.116216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.553995] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.645885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.292210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.642696] Lustre: lustre-OST0000: new disk, initializing [ 244.648254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.655958] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.761922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.590867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.606087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.656204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.991796] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.184854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.352575] Lustre: lustre-OST0001: new disk, initializing [ 268.356805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.369594] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.427540] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.700790] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.073142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.088891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.179664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 289.285953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 296.898133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.272605] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.963286] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 314.776397] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 317.261943] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 319.815620] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 321.803534] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 07:59:04 EDT 2026 [ 338.516797] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 340.156167] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.345812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.357490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.391150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.370335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.370587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.394506] Lustre: Skipped 1 previous similar message [ 358.432870] Lustre: Skipped 3 previous similar messages [ 362.160872] Lustre: server umount lustre-MDT0000 complete [ 365.850958] LustreError: 7928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523190 with bad export cookie 16849557363740573338 [ 365.861092] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.864798] LustreError: 7928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 368.609324] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.610045] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.640341] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 368.646497] Lustre: Skipped 1 previous similar message [ 368.647276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.658119] LustreError: 12469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.658417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.685039] LustreError: 12469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 370.725018] Lustre: Skipped 2 previous similar messages [ 375.783938] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.801311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.816778] Lustre: Skipped 1 previous similar message [ 380.385707] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 380.627916] Lustre: server umount lustre-MDT0001 complete [ 384.674802] Lustre: server umount lustre-OST0000 complete [ 388.092746] Lustre: server umount lustre-OST0001 complete [ 394.489822] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 405.291626] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 444.366754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.724211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.774633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.841549] Lustre: lustre-MDT0000: new disk, initializing [ 444.907120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.918739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.189882] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 462.061702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.201969] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.260841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.265151] Lustre: Skipped 1 previous similar message [ 462.317259] Lustre: lustre-MDT0001: new disk, initializing [ 462.416826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 462.456153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 462.471578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.062631] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 477.455514] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.747150] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.082255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.279476] Lustre: lustre-OST0000: new disk, initializing [ 491.282519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.289528] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.366523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.031883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 493.050376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.204090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 497.538697] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 506.177330] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.887262] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.091518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.450771] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.614428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 518.630758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.641607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 518.653926] Lustre: Skipped 1 previous similar message [ 518.655629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.656993] Lustre: Skipped 1 previous similar message [ 523.775369] LustreError: 17042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.797741] LustreError: 17042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 523.976498] Lustre: server umount lustre-OST0000 complete [ 528.865405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.874355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.878457] Lustre: Skipped 2 previous similar messages [ 528.889084] LustreError: Skipped 1 previous similar message [ 533.689615] Lustre: server umount lustre-MDT0000 complete [ 533.985679] LustreError: 17066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.024840] LustreError: 17066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 537.687834] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523362 with bad export cookie 16849557363740574871 [ 537.694805] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 537.695789] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.123787] Lustre: server umount lustre-MDT0001 complete [ 544.274380] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:02:46 (1788523366) [ 562.731954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.168381] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.243746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.799797] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 575.747485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.525617] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 587.977737] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.929888] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.114301] Lustre: Failing over lustre-MDT0000 [ 596.292970] Lustre: server umount lustre-MDT0000 complete [ 596.449957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.458232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.472071] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.485921] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 614.881241] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788523423/real 1788523423] req@ffff8f13c3d80700 x1875402235737344/t0(0) o400->MGC192.168.202.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788523439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.890126] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.010982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.121985] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f14fe304a80 x1875402235740160/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 625.364365] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 629.891847] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 630.764653] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 630.782072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 630.828994] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 639.108541] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 645.284873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.292920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.296764] Lustre: Skipped 2 previous similar messages [ 645.313754] Lustre: Skipped 3 previous similar messages [ 648.581881] Lustre: server umount lustre-MDT0000 complete [ 650.740914] LustreError: 20664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.774596] LustreError: 20664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 652.922440] LustreError: 22596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523477 with bad export cookie 16849557363740576285 [ 652.932327] LustreError: 22596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 652.936152] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.300374] Lustre: server umount lustre-MDT0001 complete [ 661.925505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.569421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.572551] Lustre: Skipped 2 previous similar messages [ 667.533549] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 674.812131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.570599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 686.716353] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.320205] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 694.413821] Lustre: Failing over lustre-MDT0000 [ 694.804258] Lustre: server umount lustre-MDT0000 complete [ 695.794402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.801958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.187869] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788523522/real 1788523522] req@ffff8f14fe329180 x1875402235767040/t0(0) o400->MGC192.168.202.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788523538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 713.219420] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.517811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.309434] LustreError: 24564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.336779] LustreError: 24564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 723.990395] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 728.865863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 729.077994] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.082395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.098276] Lustre: Skipped 1 previous similar message [ 729.113185] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 738.350650] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 743.397266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.586499] Lustre: server umount lustre-MDT0000 complete [ 751.771907] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523576 with bad export cookie 16849557363740577223 [ 751.781829] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.134414] Lustre: server umount lustre-MDT0001 complete [ 760.965413] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:06:23 (1788523583) [ 778.153271] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 790.037474] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 833.191313] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 845.606929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.906177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 845.935553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 845.995670] Lustre: lustre-MDT0000: new disk, initializing [ 846.077845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.082445] Lustre: Skipped 2 previous similar messages [ 846.096860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 850.228598] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 861.286377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.426911] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 861.842939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 866.584398] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 876.083059] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 881.810712] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 888.735238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.966727] Lustre: 31614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 890.631949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 890.737690] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 890.825910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 890.840400] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 895.938400] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 905.279642] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.704364] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 931.821264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.835794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.841923] Lustre: Skipped 3 previous similar messages [ 931.848413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.860168] Lustre: Skipped 4 previous similar messages [ 935.504230] Lustre: server umount lustre-OST0000 complete [ 954.337416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.514081] Lustre: server umount lustre-MDT0000 complete [ 956.387982] LustreError: 32339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.411834] LustreError: 32339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 958.541085] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523783 with bad export cookie 16849557363740577629 [ 958.551212] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 958.553035] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.151759] Lustre: server umount lustre-MDT0001 complete [ 969.676174] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 972.158612] Key type lgssc unregistered [ 972.422378] LNet: 33598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.451602] LNetError: 33598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.477030] LNet: Removed LNI 192.168.202.134@tcp [ 973.474151] Key type .llcrypt unregistered [ 973.476178] Key type ._llcrypt unregistered [ 998.801357] Key type ._llcrypt registered [ 998.806538] Key type .llcrypt registered [ 998.992875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1000.261321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1000.395483] alg: No test for adler32 (adler32-zlib) [ 1001.620395] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1002.030134] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1003.704969] Key type lgssc registered [ 1004.768976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.707671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1012.725598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.192964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.853151] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1023.461385] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.476960] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1027.337967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.627683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.511506] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1038.216669] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1044.085101] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1051.081881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.543632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.642852] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1057.343041] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1058.798763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1065.681937] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.331691] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:11:42 (1788523902) [ 1082.283305] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1084.332401] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1091.137724] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:11:54 (1788523914) [ 1099.382952] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:12:01 (1788523921) [ 1100.211757] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1107.971798] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1107.980200] Lustre: Skipped 2 previous similar messages [ 1109.729246] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:12:12 (1788523932) [ 1113.537770] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1113.542781] Lustre: Skipped 3 previous similar messages [ 1131.004155] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1131.015328] Lustre: Skipped 6 previous similar messages [ 1133.719486] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:12:36 (1788523956) [ 1136.632764] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1140.428135] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1149.243924] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:12:52 (1788523972) [ 1150.884818] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1150.888526] Lustre: Skipped 3 previous similar messages [ 1164.726075] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:13:07 (1788523987) [ 1180.641213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.644933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.652747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.665827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.674673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.680171] Lustre: Skipped 1 previous similar message [ 1185.503238] Lustre: server umount lustre-OST0000 complete [ 1191.393983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1191.415693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.448136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.026251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.035121] Lustre: Skipped 2 previous similar messages [ 1202.164108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.169587] Lustre: Skipped 1 previous similar message [ 1203.171973] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.338899] Lustre: server umount lustre-MDT0000 complete [ 1206.249468] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.279845] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1207.281277] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524032 with bad export cookie 12044945887236439916 [ 1207.286359] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.298565] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1207.816599] Lustre: server umount lustre-MDT0001 complete [ 1218.613764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1221.267049] Key type lgssc unregistered [ 1221.582380] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.590648] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.604449] LNet: Removed LNI 192.168.202.134@tcp [ 1222.533313] Key type .llcrypt unregistered [ 1222.536885] Key type ._llcrypt unregistered [ 1238.086903] Key type ._llcrypt registered [ 1238.089404] Key type .llcrypt registered [ 1238.164604] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 1253.188549] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1254.308223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.389863] alg: No test for adler32 (adler32-zlib) [ 1255.455083] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1255.736944] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1257.419146] Key type lgssc registered [ 1258.696868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.028202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1294.057982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.301563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1295.321487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1295.382335] Lustre: lustre-MDT0000: new disk, initializing [ 1295.488912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.511265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1298.948614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1308.507190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.632716] Lustre: 46002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1308.693161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1308.696964] Lustre: Skipped 1 previous similar message [ 1308.764202] Lustre: lustre-MDT0001: new disk, initializing [ 1308.866095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.931751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1308.952625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1313.737512] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1322.651274] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.881247] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1335.815517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.146650] Lustre: lustre-OST0000: new disk, initializing [ 1336.149657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1336.153657] Lustre: 47971:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1336.220906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.147164] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1346.144397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1346.150087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1346.248789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1351.177251] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1355.942304] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1356.147336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1361.286536] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1361.492667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1367.017574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.020752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.026218] LustreError: Skipped 1 previous similar message [ 1367.027223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.057687] Lustre: Skipped 1 previous similar message [ 1369.660824] Lustre: server umount lustre-OST0000 complete [ 1375.714583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.725870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.749411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.761706] Lustre: Skipped 1 previous similar message [ 1377.256064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.257320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.329770] Lustre: server umount lustre-MDT0000 complete [ 1384.030931] LustreError: 45995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524208 with bad export cookie 14851241530160670489 [ 1384.038341] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.042710] LustreError: 45995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1384.376183] Lustre: server umount lustre-MDT0001 complete [ 1394.140420] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:16:56 (1788524216) [ 1401.037054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.529350] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.541329] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1401.616475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.548687] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1406.952665] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.068902] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.431087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.857796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.582513] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1428.172405] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1433.961698] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.889072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.191455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.595285] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1455.278436] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.131272] LustreError: 52969:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1465.166978] Lustre: Modifying parameter general.timeout=1 in log params [ 1466.892284] Lustre: Modifying parameter general.timeout=3 in log params [ 1466.896119] Lustre: Skipped 1 previous similar message [ 1469.363914] Lustre: Modifying parameter general.timeout=6 in log params [ 1469.370045] Lustre: Skipped 2 previous similar messages [ 1474.244526] Lustre: Modifying parameter general.timeout=11 in log params [ 1474.252358] Lustre: Skipped 4 previous similar messages [ 1482.802166] Lustre: Modifying parameter general.timeout=21 in log params [ 1482.809096] Lustre: Skipped 9 previous similar messages [ 1499.141738] Lustre: Modifying parameter general.timeout=41 in log params [ 1499.146137] Lustre: Skipped 19 previous similar messages [ 1531.647675] Lustre: Modifying parameter general.timeout=83 in log params [ 1531.650319] Lustre: Skipped 41 previous similar messages [ 1577.681816] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:20:00 (1788524400) [ 1580.789551] Lustre: 59909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1582.486331] Lustre: 60008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1585.960190] Lustre: 60204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1585.969287] Lustre: 60204:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1595.699898] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1595.710353] Lustre: Skipped 58 previous similar messages [ 1618.437066] Lustre: 61779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1618.445279] Lustre: 61779:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1618.464491] LustreError: 61779:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1622.643630] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:20:45 (1788524445) [ 1635.812132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.819249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.827121] Lustre: Skipped 1 previous similar message [ 1635.846960] Lustre: Skipped 3 previous similar messages [ 1640.932746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.941619] Lustre: Skipped 2 previous similar messages [ 1641.992128] Lustre: server umount lustre-MDT0000 complete [ 1645.866452] LustreError: 51177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524470 with bad export cookie 14851241530160671371 [ 1645.882978] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.888459] LustreError: 51177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1646.050045] LustreError: 50437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.054212] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.074415] Lustre: Skipped 1 previous similar message [ 1646.187478] Lustre: server umount lustre-MDT0001 complete [ 1652.561663] Lustre: server umount lustre-OST0000 complete [ 1679.544380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.884802] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1680.127669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.621621] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1693.582349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.681935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1693.705720] Lustre: 64311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1698.403228] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1706.399928] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1712.951661] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.635925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.915274] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1721.264665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.267276] Lustre: Skipped 1 previous similar message [ 1726.831689] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1734.791093] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1736.462205] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1736.470760] Lustre: Skipped 11 previous similar messages [ 1736.546034] Lustre: 66879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1745.891598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.903186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.924889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1750.611841] Lustre: server umount lustre-OST0000 complete [ 1755.616910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.624156] LustreError: Skipped 1 previous similar message [ 1760.988766] Lustre: server umount lustre-MDT0000 complete [ 1761.763028] LustreError: 64320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.789402] LustreError: 64320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1765.959682] LustreError: 66175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524590 with bad export cookie 14851241530160674283 [ 1765.967444] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.974644] LustreError: 66175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1766.470850] Lustre: server umount lustre-MDT0001 complete [ 1777.163562] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1779.960368] Key type lgssc unregistered [ 1780.284689] LNet: 68089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.288778] LNetError: 68089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.302904] LNet: Removed LNI 192.168.202.134@tcp [ 1781.164416] Key type .llcrypt unregistered [ 1781.165373] Key type ._llcrypt unregistered [ 1803.805088] Key type ._llcrypt registered [ 1803.814255] Key type .llcrypt registered [ 1803.926928] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1805.037446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.056921] alg: No test for adler32 (adler32-zlib) [ 1806.237745] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1806.650368] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1808.432378] Key type lgssc registered [ 1809.876736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.872584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.914709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.697600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.274543] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1831.907276] LustreError: 69499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1831.956340] LustreError: 69499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1837.052136] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.108056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.991509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.155614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1856.160832] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1862.641766] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1870.787416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1871.742480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.476569] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1888.590952] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1902.019453] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1911.778966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.783571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.804630] Lustre: Skipped 2 previous similar messages [ 1911.808451] Lustre: Skipped 1 previous similar message [ 1916.898486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1916.911510] Lustre: Skipped 3 previous similar messages [ 1917.905996] Lustre: server umount lustre-MDT0000 complete [ 1921.453878] LustreError: 70240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524746 with bad export cookie 11748723615606800439 [ 1921.460209] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1921.465734] LustreError: 70240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1921.755788] Lustre: server umount lustre-MDT0001 complete [ 1936.170154] Lustre: server umount lustre-OST0000 complete [ 1964.907758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.271774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1965.310885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1965.671515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.048757] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1981.164242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.254363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1981.273685] Lustre: 74373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1981.868114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.122923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1995.114949] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.711209] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2008.414743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2008.647459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2009.101172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.868488] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2024.562391] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2026.646901] Lustre: 76949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.637770] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2037.730398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.744876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.769158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.867876] Lustre: server umount lustre-OST0000 complete [ 2047.970377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2047.991757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.028488] Lustre: Skipped 1 previous similar message [ 2048.031851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2048.053906] Lustre: Skipped 2 previous similar messages [ 2052.047095] Lustre: server umount lustre-MDT0000 complete [ 2053.601701] LustreError: 74386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.635611] LustreError: 74386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2056.320263] LustreError: 76955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524881 with bad export cookie 11748723615606801293 [ 2056.321515] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.340871] LustreError: 76955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.563153] Lustre: server umount lustre-MDT0001 complete [ 2067.841033] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2070.712972] Key type lgssc unregistered [ 2071.117497] LNet: 78163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.126446] LNetError: 78163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.157953] LNet: Removed LNI 192.168.202.134@tcp [ 2072.406223] Key type .llcrypt unregistered [ 2072.410996] Key type ._llcrypt unregistered [ 2100.698355] Key type ._llcrypt registered [ 2100.703484] Key type .llcrypt registered [ 2100.874599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2102.524565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.571720] alg: No test for adler32 (adler32-zlib) [ 2103.693522] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2104.039262] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2105.728868] Key type lgssc registered [ 2107.073275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.943495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.958342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.490309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.915614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2126.818557] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.831547] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2131.246457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.795739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.021953] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2144.326462] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.745915] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2158.317738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.905264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.959822] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2173.682239] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2183.567192] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2184.638392] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2186.094692] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2186.099842] Lustre: Skipped 1 previous similar message [ 2188.462863] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2188.469075] Lustre: Skipped 2 previous similar messages [ 2192.945613] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2192.961495] Lustre: Skipped 4 previous similar messages [ 2201.440876] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2201.446146] Lustre: Skipped 8 previous similar messages [ 2217.540899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2217.543524] Lustre: Skipped 17 previous similar messages [ 2250.373636] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2250.381949] Lustre: Skipped 32 previous similar messages [ 2287.145856] Lustre: 87677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2290.195965] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:31:52 (1788525112) [ 2292.031428] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2294.152949] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:31:56 (1788525116) [ 2300.388401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.391524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.407242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.438277] Lustre: Skipped 1 previous similar message [ 2305.514137] LustreError: 81400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.532820] LustreError: 81400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2305.667033] Lustre: server umount lustre-OST0000 complete [ 2310.626797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.638798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.644204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.659942] Lustre: Skipped 2 previous similar messages [ 2315.750132] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.774675] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2315.809588] Lustre: server umount lustre-MDT0000 complete [ 2320.126995] LustreError: 79554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525144 with bad export cookie 1304443566668060917 [ 2320.134821] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.449384] Lustre: server umount lustre-MDT0001 complete [ 2330.724932] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2333.507936] Key type lgssc unregistered [ 2333.708817] LNet: 88924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.717258] LNetError: 88924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.762457] LNet: Removed LNI 192.168.202.134@tcp [ 2335.692745] Key type .llcrypt unregistered [ 2335.695683] Key type ._llcrypt unregistered [ 2340.800736] Key type ._llcrypt registered [ 2340.802687] Key type .llcrypt registered [ 2340.912591] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2346.729545] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules [ 2348.000839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.016337] alg: No test for adler32 (adler32-zlib) [ 2349.286857] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2349.770642] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2351.473376] Key type lgssc registered [ 2352.878696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2354.173725] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2375.132618] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2377.459921] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2384.996246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.015429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.416926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.849922] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2396.642148] LustreError: 91195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.675190] LustreError: 91195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2398.885138] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:33:41 (1788525221) [ 2401.768280] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.523663] LustreError: 91226:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2403.535467] LustreError: 91226:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2403.540354] LustreError: 91226:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2403.829675] Lustre: server umount lustre-MDT0000 complete [ 2418.117705] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2420.994812] Key type lgssc unregistered [ 2421.337781] LNet: 92721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.346927] LNetError: 92721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.364407] LNet: Removed LNI 192.168.202.134@tcp [ 2422.373575] Key type .llcrypt unregistered [ 2422.386100] Key type ._llcrypt unregistered [ 2442.932975] Key type ._llcrypt registered [ 2442.938189] Key type .llcrypt registered [ 2443.075305] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2444.615310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2445.064409] alg: No test for adler32 (adler32-zlib) [ 2446.303636] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2446.636853] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2448.393623] Key type lgssc registered [ 2449.763657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.352462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.381340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.910082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.549881] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2469.347546] LustreError: 93937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.369778] LustreError: 93937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2473.934337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.874570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2480.055068] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2489.096710] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2495.313084] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2501.806377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.276460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.388926] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2516.771816] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.865142] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:36:27 (1788525387) [ 2566.527793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2568.667523] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:36:31 (1788525391) [ 2576.380466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.393420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.401170] Lustre: Skipped 2 previous similar messages [ 2576.414093] Lustre: Skipped 2 previous similar messages [ 2578.426369] Lustre: server umount lustre-MDT0000 complete [ 2581.475464] LustreError: 94668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.491771] LustreError: 94668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2582.262169] LustreError: 93919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525407 with bad export cookie 8354893709263201957 [ 2582.263493] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.274053] LustreError: 93919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2582.543609] Lustre: server umount lustre-MDT0001 complete [ 2597.186210] Lustre: server umount lustre-OST0000 complete [ 2609.446569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.967702] LustreError: 98223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.239872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.265743] LustreError: 98224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.300880] LustreError: 98224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2616.011855] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2619.374481] LustreError: 98223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.492845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.392119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.577321] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2639.450375] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.600625] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.622535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.893762] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2659.904721] LustreError: 98211:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2659.928348] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2659.934709] LustreError: 100307:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2659.944636] LustreError: 100307:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2659.954623] LustreError: 100307:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2660.031990] Lustre: server umount lustre-OST0000 complete [ 2660.039516] LustreError: 100307:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2670.380120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.508740] LustreError: 100780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.539782] LustreError: 100780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2670.728819] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2670.747448] LustreError: 98212:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2670.795543] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2670.808296] LustreError: 100775:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2670.820679] LustreError: 100775:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2670.830313] LustreError: 100775:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2670.915574] Lustre: server umount lustre-OST0000 complete [ 2670.926504] LustreError: 100775:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2681.249457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.429201] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2681.437165] Lustre: Found index 0 for lustre-OST0000, updating log [ 2681.447864] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2681.509284] Lustre: lustre-OST0000: new disk, initializing [ 2681.512565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2681.525604] Lustre: 101310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2681.819750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.377040] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2693.292994] Lustre: Failing over lustre-OST0000 [ 2693.519886] Lustre: server umount lustre-OST0000 complete [ 2700.900658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.912984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.487825] Lustre: server umount lustre-MDT0000 complete [ 2706.403897] LustreError: 98223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.424825] LustreError: 98223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2708.084248] LustreError: 98205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525532 with bad export cookie 8354893709263204694 [ 2708.089686] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.093059] LustreError: 98205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2708.407446] Lustre: server umount lustre-MDT0001 complete [ 2716.976424] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:38:59 (1788525539) [ 2735.765079] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2748.561760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.046856] LustreError: 103655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.295931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.829828] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2763.724172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.986982] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2772.029269] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2778.185524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.800748] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2795.222304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.357477] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2795.514829] Lustre: lustre-OST0001: new disk, initializing [ 2795.522674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2795.529865] Lustre: 105974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2795.888489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.896891] Lustre: Skipped 2 previous similar messages [ 2803.404659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2803.429302] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2803.631684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2804.158593] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2812.792517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2821.146726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2828.159834] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 2828.173706] Lustre: Skipped 1 previous similar message [ 2829.291000] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.316133] Lustre: Skipped 1 previous similar message [ 2832.934440] Lustre: server umount lustre-MDT0001 complete [ 2833.248033] LustreError: 103652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.278322] LustreError: 103652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2843.383200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.490058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2843.495263] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2843.541634] Lustre: 103643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2845.552204] LustreError: 107547:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2845.568963] LustreError: 107547:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2845.580253] LustreError: 103647:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2845.735351] LustreError: 107558:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -17 [ 2845.741856] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2845.741856] [ 2849.291726] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2849.360114] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2859.254662] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:41:22 (1788525682) [ 2878.512473] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2897.738709] Lustre: 109588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2920.734533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2923.848448] Lustre: 110723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3007.973267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.975141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.979267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.979276] Lustre: Skipped 3 previous similar messages [ 3008.003028] Lustre: Skipped 4 previous similar messages [ 3010.268259] Lustre: server umount lustre-MDT0000 complete [ 3013.093023] LustreError: 103651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.125840] LustreError: 103651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3014.068311] LustreError: 103635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525838 with bad export cookie 8354893709263205695 [ 3014.076701] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.081553] LustreError: 103635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3014.299627] Lustre: server umount lustre-MDT0001 complete [ 3027.617775] Lustre: server umount lustre-OST0000 complete [ 3041.974967] Lustre: server umount lustre-OST0001 complete [ 3048.426524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3059.403066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3069.386156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3080.291474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3101.940428] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3113.038850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.056269] Lustre: lustre-MDT0000: reset Object Index mappings [ 3113.797116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.799909] Lustre: Skipped 1 previous similar message [ 3118.263836] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3128.808652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.828454] Lustre: lustre-MDT0001: reset Object Index mappings [ 3135.460782] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3138.893667] Lustre: 115822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3145.703947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.454293] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3154.642783] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3158.736126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3167.507777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.898587] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3173.904369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3175.702505] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3184.480074] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3188.638753] Lustre: 117700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3244.943551] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:47:47 (1788526067) [ 3271.141061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.170293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.188133] Lustre: Skipped 1 previous similar message [ 3271.208616] Lustre: Skipped 5 previous similar messages [ 3276.863424] Lustre: server umount lustre-MDT0000 complete [ 3281.387941] LustreError: 116199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.419600] LustreError: 116199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3282.237040] LustreError: 115424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788526107 with bad export cookie 8354893709263252546 [ 3282.251320] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.254070] LustreError: 115424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3286.500898] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.514817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3286.535841] Lustre: Skipped 3 previous similar messages [ 3286.565888] Lustre: Skipped 5 previous similar messages [ 3289.181217] Lustre: server umount lustre-MDT0001 complete [ 3300.066840] Lustre: server umount lustre-OST0000 complete [ 3311.071868] Lustre: server umount lustre-OST0001 complete [ 3318.787814] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3331.428565] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3375.335988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.690125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3375.747349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3375.867638] Lustre: lustre-MDT0000: new disk, initializing [ 3375.945764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.955718] Lustre: Skipped 3 previous similar messages [ 3376.009527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3381.331170] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3389.090615] Lustre: server umount lustre-MDT0000 complete [ 3399.917781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.348465] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3400.366609] Lustre: 121975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3400.374249] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3406.110741] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3411.776591] Lustre: server umount lustre-MDT0000 complete [ 3421.893622] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:50:44 (1788526244) [ 3423.933648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3426.157412] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:50:48 (1788526248) [ 3445.448406] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3457.634379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.933705] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3474.107844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.310315] Lustre: 124025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3474.467832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3474.478690] Lustre: Skipped 1 previous similar message [ 3474.618121] Lustre: lustre-MDT0001: new disk, initializing [ 3474.928149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3474.952894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3480.711348] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3486.258896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3493.139952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3493.355654] Lustre: lustre-OST0000: new disk, initializing [ 3493.362801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.369512] Lustre: 125603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3494.608231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3494.641388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3494.847579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3501.401429] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3514.111050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.396733] Lustre: lustre-OST0001: new disk, initializing [ 3514.403634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3514.414969] Lustre: 126473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3515.585468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3515.601705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3515.749628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3522.652920] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3534.266328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3539.193172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3961.828459] Lustre: 132898:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3961.835866] Lustre: 132898:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3961.838995] Lustre: 132898:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3961.847709] Lustre: 132898:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3961.854891] Lustre: 132898:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3961.867550] Lustre: 132898:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4234.719616] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:04:16 (1788527056) [ 4253.154876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4253.164768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.168230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.168239] Lustre: Skipped 2 previous similar messages [ 4253.221447] Lustre: Skipped 2 previous similar messages [ 4256.724447] Lustre: server umount lustre-MDT0000 complete [ 4258.279341] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.295812] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4260.485691] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788527085 with bad export cookie 8354893709263321447 [ 4260.493123] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.504986] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4263.395085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.396503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4263.403339] Lustre: Skipped 2 previous similar messages [ 4263.411248] Lustre: Skipped 3 previous similar messages [ 4267.171708] Lustre: server umount lustre-MDT0001 complete [ 4285.408207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4285.627703] Lustre: server umount lustre-OST0000 complete [ 4295.570361] Lustre: server umount lustre-OST0001 complete [ 4301.989302] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4313.633467] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4352.413671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.713308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4352.752145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4352.876632] Lustre: lustre-MDT0000: new disk, initializing [ 4353.026083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.032970] Lustre: Skipped 5 previous similar messages [ 4353.086344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4357.875613] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4368.712963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.840160] Lustre: 135967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.911283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4368.915812] Lustre: Skipped 1 previous similar message [ 4368.991701] Lustre: lustre-MDT0001: new disk, initializing [ 4369.140188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4369.148589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4374.140857] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4384.128908] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.538507] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4395.874034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.182679] Lustre: lustre-OST0000: new disk, initializing [ 4396.185607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4396.190795] Lustre: 137933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4396.285744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.288138] Lustre: Skipped 1 previous similar message [ 4398.024386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4398.039501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4398.157712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4402.278463] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4412.585779] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4423.647668] Lustre: lustre-MDD0000: changelog on [ 4426.523329] Lustre: lustre-MDD0001: changelog on [ 6109.420422] Lustre: lustre-MDD0001: changelog off [ 6111.157567] Lustre: 135972:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6111.907211] Lustre: lustre-MDD0000: changelog off [ 6118.372845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.374329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.383833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6118.383842] Lustre: Skipped 3 previous similar messages [ 6118.422202] Lustre: Skipped 1 previous similar message [ 6122.886808] Lustre: server umount lustre-OST0000 complete [ 6128.626689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.631836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6128.643701] Lustre: Skipped 1 previous similar message [ 6128.660469] Lustre: Skipped 2 previous similar messages [ 6133.731067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.853397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.868297] Lustre: Skipped 2 previous similar messages [ 6140.384143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6140.790588] Lustre: server umount lustre-MDT0000 complete [ 6143.970878] LustreError: 135978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.985389] LustreError: 135978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6144.207547] LustreError: 135959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788528969 with bad export cookie 8354893709264139705 [ 6144.218318] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.231746] LustreError: 135959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6144.569775] Lustre: server umount lustre-MDT0001 complete [ 6155.776952] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6159.387708] Key type lgssc unregistered [ 6159.675499] LNet: 147024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.688573] LNetError: 147024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6159.705541] LNet: Removed LNI 192.168.202.134@tcp [ 6160.727289] Key type .llcrypt unregistered [ 6160.731990] Key type ._llcrypt unregistered [ 6179.827655] Key type ._llcrypt registered [ 6179.830121] Key type .llcrypt registered [ 6179.916702] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6192.972352] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6193.792452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.903939] alg: No test for adler32 (adler32-zlib) [ 6195.028912] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 6195.216211] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6196.848173] Key type lgssc registered [ 6197.975430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6233.550192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6233.596452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.912730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6234.953399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6235.052846] Lustre: lustre-MDT0000: new disk, initializing [ 6235.131930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.145423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6239.689395] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6249.845853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.912058] Lustre: 150202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6249.949763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6249.952786] Lustre: Skipped 1 previous similar message [ 6250.032563] Lustre: lustre-MDT0001: new disk, initializing [ 6250.238145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6250.311637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6250.333770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6255.604234] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6264.415967] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6271.024573] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6278.528235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.793202] Lustre: lustre-OST0000: new disk, initializing [ 6278.798609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6278.808785] Lustre: 152170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6278.899191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.047466] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6288.474435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6288.479481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6288.604603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6294.797542] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.715041] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_module obdecho/obdecho [ 6301.209960] LustreError: 153171:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6301.215847] LustreError: 153171:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6302.632795] LustreError: 153267:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6314.466684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.470609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6314.474990] Lustre: Skipped 1 previous similar message [ 6319.606665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.611660] Lustre: Skipped 3 previous similar messages [ 6319.883311] Lustre: server umount lustre-MDT0000 complete [ 6323.200886] LustreError: 150194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529148 with bad export cookie 11053576920896291224 [ 6323.202225] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6323.209030] LustreError: 150194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6323.636781] Lustre: server umount lustre-MDT0001 complete [ 6338.092879] Lustre: server umount lustre-OST0000 complete [ 6360.035961] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6369.504182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6408.113209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.431565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6408.450963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6408.487203] Lustre: lustre-MDT0000: new disk, initializing [ 6408.546348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.562378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6413.450975] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6425.177143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.339786] Lustre: 156986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6425.418360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6425.423186] Lustre: Skipped 1 previous similar message [ 6425.519804] Lustre: lustre-MDT0001: new disk, initializing [ 6425.690199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.759903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6425.769215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6431.342505] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6441.277693] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.622981] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6454.416535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.633241] Lustre: lustre-OST0000: new disk, initializing [ 6454.636355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6454.641291] Lustre: 158953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6454.724664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.489327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6460.496996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6460.637130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6462.251382] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6472.345766] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6477.492341] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6477.820801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6481.785607] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6481.935605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6486.512517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.513275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.527984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6486.549048] Lustre: Skipped 2 previous similar messages [ 6486.554373] LustreError: Skipped 1 previous similar message [ 6489.671786] Lustre: server umount lustre-OST0000 complete [ 6496.745777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.747616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.767111] Lustre: Skipped 1 previous similar message [ 6496.776684] Lustre: Skipped 1 previous similar message [ 6500.850776] Lustre: server umount lustre-MDT0000 complete [ 6504.668282] LustreError: 156978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529329 with bad export cookie 11053576920896292183 [ 6504.678448] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6504.680928] LustreError: 156978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6505.169983] Lustre: server umount lustre-MDT0001 complete [ 6508.828117] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:42:11 (1788529331) [ 6510.457224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6512.504926] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:42:15 (1788529335) [ 6521.375311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.019233] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.037727] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6522.088469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6526.040786] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6527.460279] LustreError: 161369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.562927] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.683199] LustreError: 161369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6539.151420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.678657] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6557.910607] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6563.770488] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6570.404358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.766418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.773199] Lustre: Skipped 1 previous similar message [ 6577.249568] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6585.441595] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.186727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6601.201562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.234237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.246684] Lustre: Skipped 1 previous similar message [ 6603.259593] Lustre: server umount lustre-MDT0000 complete [ 6606.689885] LustreError: 161364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.021692] LustreError: 162103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529431 with bad export cookie 11053576920896293065 [ 6607.026087] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6607.031499] LustreError: 162103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6611.841802] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 6611.852165] Lustre: Skipped 3 previous similar messages [ 6611.941928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.962498] Lustre: Skipped 2 previous similar messages [ 6613.488933] Lustre: server umount lustre-MDT0001 complete [ 6619.689420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6633.386141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.597912] LustreError: 165231:0:(llog.c:1655:llog_backup()) MGC192.168.202.134@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6633.605193] Lustre: 165231:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.134@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6636.514781] LustreError: 165236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.537454] LustreError: 165236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6653.781736] LustreError: 165238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.805529] LustreError: 165238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6686.690166] LustreError: 165238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.724956] LustreError: 165238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6694.369403] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6694.375224] Lustre: 165231:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6735.365040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.499496] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6740.973124] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6750.697776] LustreError: 163201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.731337] LustreError: 163201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6751.910270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.086626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 6758.394791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6758.726943] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6768.147156] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.860045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6778.511897] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6780.461104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6803.505951] Lustre: server umount lustre-MDT0000 complete [ 6804.451340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6804.467856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6804.470087] Lustre: Skipped 1 previous similar message [ 6808.579403] LustreError: 161350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529633 with bad export cookie 11053576920896299519 [ 6808.581363] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.607810] LustreError: 161350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6809.233754] Lustre: server umount lustre-MDT0001 complete [ 6825.196427] Lustre: server umount lustre-OST0000 complete [ 6852.829418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6869.597569] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6918.502511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.822967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6918.866491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6918.948461] Lustre: lustre-MDT0000: new disk, initializing [ 6919.078038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.091743] Lustre: Skipped 1 previous similar message [ 6919.114859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6925.342877] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6938.772866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.860429] Lustre: 170361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6939.146769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6945.073370] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6956.558333] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6963.673579] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6974.801578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.410404] Lustre: lustre-OST0000: new disk, initializing [ 6975.420859] Lustre: Skipped 1 previous similar message [ 6975.442074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6975.452422] Lustre: Skipped 2 previous similar messages [ 6975.467257] Lustre: 172331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6983.457369] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6984.895286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6984.904512] Lustre: Skipped 1 previous similar message [ 6984.921570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6985.052284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6995.740581] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7001.727182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7002.019571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7008.415618] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7008.922145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7015.913616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7015.925498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.941596] Lustre: Skipped 1 previous similar message [ 7015.960678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7015.968734] Lustre: Skipped 3 previous similar messages [ 7026.144229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7026.444676] Lustre: server umount lustre-OST0000 complete [ 7036.386495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7036.393770] LustreError: Skipped 1 previous similar message [ 7038.038125] Lustre: server umount lustre-MDT0000 complete [ 7041.510045] LustreError: 171362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.537798] LustreError: 171362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7042.990457] LustreError: 171155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788529867 with bad export cookie 11053576920896316347 [ 7042.994124] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7042.999387] LustreError: 171155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7043.470303] Lustre: server umount lustre-MDT0001 complete [ 7054.594532] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:51:16 (1788529876) [ 7062.435687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.673126] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7081.377783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.287342] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7097.564791] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7104.912990] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.827016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7122.896675] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7134.635487] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.070712] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:52:51 (1788529971) [ 7154.665917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7154.676715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.693766] Lustre: Skipped 4 previous similar messages [ 7154.695691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7154.725583] Lustre: Skipped 7 previous similar messages [ 7160.798475] Lustre: server umount lustre-OST0000 complete [ 7172.869470] Lustre: server umount lustre-MDT0000 complete [ 7177.952676] LustreError: 174723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530002 with bad export cookie 11053576920896317229 [ 7177.978285] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.993802] LustreError: 174723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7178.616911] Lustre: server umount lustre-MDT0001 complete [ 7193.740573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7197.804856] Key type lgssc unregistered [ 7198.300402] LNet: 178476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7198.310589] LNetError: 178476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7199.337463] LNet: Removed LNI 192.168.202.134@tcp [ 7200.713523] Key type .llcrypt unregistered [ 7200.717818] Key type ._llcrypt unregistered [ 7225.663491] Key type ._llcrypt registered [ 7225.666402] Key type .llcrypt registered [ 7225.831567] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7227.487274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7227.520150] alg: No test for adler32 (adler32-zlib) [ 7228.936188] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 7229.327190] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7231.032189] Key type lgssc registered [ 7232.259334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7241.503684] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7241.540418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.249355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7262.368845] LustreError: 179717:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 7267.553374] LustreError: 179717:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7267.565818] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7267.573196] LustreError: MGC192.168.202.134@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7267.580756] LustreError: 179717:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7267.587879] LustreError: 179717:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7267.593478] LustreError: 179717:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7267.708709] Lustre: server umount lustre-OST0000 complete [ 7267.732922] LustreError: 179717:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7280.389219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.145854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.826976] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7291.382861] LustreError: 180217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7291.414146] LustreError: 180217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7296.494342] LustreError: 180216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.007844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.395420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7305.650941] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7315.398616] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7322.671230] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7331.878622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.114289] LustreError: 182034:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7332.126948] Lustre: 182034:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.134@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7332.379851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7340.777592] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7350.872231] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.876065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7355.891819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.929152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7356.898379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.911397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7356.924339] Lustre: Skipped 1 previous similar message [ 7356.952903] Lustre: Skipped 2 previous similar messages [ 7360.303960] Lustre: server umount lustre-MDT0000 complete [ 7362.019565] LustreError: 180217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.053354] LustreError: 180217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7365.276057] LustreError: 180198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530190 with bad export cookie 16717363538206449091 [ 7365.278959] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.289687] LustreError: 180198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7365.668313] Lustre: server umount lustre-MDT0001 complete [ 7380.803778] Lustre: server umount lustre-OST0000 complete [ 7396.367992] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7412.538477] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7460.424182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.747585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7460.788196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7460.892396] Lustre: lustre-MDT0000: new disk, initializing [ 7460.994268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.012065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7467.450107] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7481.152938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.329281] Lustre: 185582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7481.455993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7481.461169] Lustre: Skipped 1 previous similar message [ 7481.537312] Lustre: lustre-MDT0001: new disk, initializing [ 7481.667275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.696658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7481.710318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7486.198700] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7498.234782] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7504.811674] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7512.881589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7513.275079] Lustre: lustre-OST0000: new disk, initializing [ 7513.279313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7513.286602] Lustre: 187551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7513.444590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.360865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7521.380584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7521.507367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7521.800973] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7533.947108] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7539.868680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7540.172926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7545.927978] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7546.210602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7552.481417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7552.484241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.493216] LustreError: Skipped 1 previous similar message [ 7552.494899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7552.530157] Lustre: Skipped 1 previous similar message [ 7557.604712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.620661] Lustre: Skipped 2 previous similar messages [ 7562.735507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7563.233833] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7563.623375] Lustre: server umount lustre-OST0000 complete [ 7572.962807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.014935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7573.020930] Lustre: Skipped 2 previous similar messages [ 7575.693743] Lustre: server umount lustre-MDT0000 complete [ 7578.097984] LustreError: 185590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.128651] LustreError: 185590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7580.769211] LustreError: 186388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530405 with bad export cookie 16717363538206449812 [ 7580.779645] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7581.158992] Lustre: server umount lustre-MDT0001 complete [ 7592.883161] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:00:14 (1788530414) [ 7614.602371] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7618.570130] Key type lgssc unregistered [ 7619.058577] LNet: 190501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7619.078154] LNetError: 190501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7619.109291] LNet: Removed LNI 192.168.202.134@tcp [ 7620.256107] Key type .llcrypt unregistered [ 7620.260485] Key type ._llcrypt unregistered [ 7646.440568] Key type ._llcrypt registered [ 7646.442843] Key type .llcrypt registered [ 7646.573549] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7648.277813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7648.326769] alg: No test for adler32 (adler32-zlib) [ 7649.487816] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 7649.753911] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7651.472258] Key type lgssc registered [ 7652.938862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7663.577133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7663.602964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.132291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.657991] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7674.341842] LustreError: 191731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.364209] LustreError: 191731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7679.461328] LustreError: 191730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.321594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.823046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7687.253177] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7697.305591] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7704.489190] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7714.116927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.436344] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7714.440166] LustreError: 193596:0:(llog.c:1696:llog_backup()) MGC192.168.202.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7714.454709] Lustre: 193596:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.134@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7714.666847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7721.488153] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7728.600246] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7732.195956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.203974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.218313] Lustre: Skipped 1 previous similar message [ 7732.236992] Lustre: Skipped 2 previous similar messages [ 7737.121220] Lustre: server umount lustre-MDT0000 complete [ 7737.331785] LustreError: 191725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.373957] LustreError: 191725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7741.284169] LustreError: 191712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530566 with bad export cookie 6245569402731755663 [ 7741.292814] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.295013] LustreError: 191712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7741.649378] Lustre: server umount lustre-MDT0001 complete [ 7757.285995] Lustre: server umount lustre-OST0000 complete [ 7769.480825] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7780.530646] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7819.099355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.342217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7819.373616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7819.461833] Lustre: lustre-MDT0000: new disk, initializing [ 7819.559310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.586065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7824.515612] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7835.222280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.348927] Lustre: 197124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7835.402643] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7835.405611] Lustre: Skipped 1 previous similar message [ 7835.408672] LustreError: 197906:0:(llog.c:1696:llog_backup()) MGC192.168.202.134@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7835.415902] LustreError: 197906:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7835.423963] Lustre: 197906:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.134@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7835.446046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7835.451495] Lustre: Skipped 1 previous similar message [ 7835.588839] Lustre: lustre-MDT0001: new disk, initializing [ 7835.749042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7835.812154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7835.825813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7841.414580] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7851.302945] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7857.609777] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7864.733697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7865.048486] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7865.051234] Lustre: Skipped 1 previous similar message [ 7865.054908] LustreError: 199068:0:(llog.c:1696:llog_backup()) MGC192.168.202.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7865.064690] LustreError: 199068:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7865.070497] Lustre: 199068:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.134@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7865.078773] Lustre: 199068:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7865.097747] Lustre: lustre-OST0000: new disk, initializing [ 7865.102207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7865.108351] Lustre: 199092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7865.210578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.583563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7871.605656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7871.776620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7872.183068] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7882.365590] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7887.447636] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7887.664438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7893.235597] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7893.431691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7897.570248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7897.578969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.585986] LustreError: Skipped 1 previous similar message [ 7897.587899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.626206] Lustre: Skipped 2 previous similar messages [ 7901.984922] Lustre: server umount lustre-OST0000 complete [ 7907.297096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7907.301480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.317148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.328150] Lustre: Skipped 1 previous similar message [ 7912.940933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7912.947094] Lustre: Skipped 2 previous similar messages [ 7913.299518] Lustre: server umount lustre-MDT0000 complete [ 7917.164085] LustreError: 197117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530741 with bad export cookie 6245569402731756377 [ 7917.168425] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.175716] LustreError: 197117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7917.569567] Lustre: server umount lustre-MDT0001 complete [ 7927.770418] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:05:50 (1788530750) [ 7945.910808] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7957.277213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.657944] LustreError: 202219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.711289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.908522] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7963.112627] LustreError: 202220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.024590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.322924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7977.927588] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7981.936483] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7990.462838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.828101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.833788] Lustre: Skipped 1 previous similar message [ 7998.134944] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8007.633856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.747829] Lustre: lustre-OST0001: new disk, initializing [ 8007.751297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8007.756618] Lustre: 204528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8014.220618] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8014.905423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8014.909905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8015.068636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8024.891208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8036.244861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8038.741478] Lustre: DEBUG MARKER: ADD OST3 [ 8044.253651] loop: module loaded [ 8044.365498] loop0: detected capacity change from 0 to 204800000 [ 8044.498810] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8044.508441] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8044.515147] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8051.307712] loop0: detected capacity change from 0 to 204800000 [ 8052.104446] Lustre: DEBUG MARKER: STOP OST3 [ 8058.067161] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8058.405223] Lustre: lustre-OST0003: new disk, initializing [ 8058.420738] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8058.440587] Lustre: 206548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8058.646818] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8058.651439] Lustre: Skipped 1 previous similar message [ 8060.341566] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8060.566918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8060.585469] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8060.637592] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8062.282966] Lustre: Failing over lustre-OST0003 [ 8062.375236] LustreError: 206702:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8062.440234] Lustre: server umount lustre-OST0003 complete [ 8065.517098] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.525268] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.525280] Lustre: Skipped 1 previous similar message [ 8065.534154] LustreError: 203703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.567424] LustreError: 203703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8066.494817] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8075.354598] Lustre: DEBUG MARKER: START OST3 again [ 8081.796369] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8082.201215] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8083.499311] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8083.918129] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8083.926460] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8089.700543] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8111.186501] Lustre: Failing over lustre-OST0003 [ 8111.277808] Lustre: server umount lustre-OST0003 complete [ 8114.659503] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.659844] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.668201] LustreError: Skipped 1 previous similar message [ 8114.672133] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.672144] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8114.755225] Lustre: Skipped 2 previous similar messages [ 8136.167361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.173178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.189211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.692813] Lustre: server umount lustre-MDT0000 complete [ 8143.640599] LustreError: 202199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788530968 with bad export cookie 6245569402731757259 [ 8143.644420] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8143.657166] LustreError: 202199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8146.917938] LustreError: 203953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.944472] LustreError: 203953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 8150.437969] Lustre: server umount lustre-MDT0001 complete [ 8154.913478] Lustre: server umount lustre-OST0000 complete [ 8159.421310] Lustre: server umount lustre-OST0001 complete [ 8166.772691] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 8176.520536] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8217.527323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.782563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8217.805949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8217.879954] Lustre: lustre-MDT0000: new disk, initializing [ 8217.947603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.950687] Lustre: Skipped 1 previous similar message [ 8217.974314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8223.408825] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8234.820974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.021571] Lustre: 211991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8235.084608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8235.094437] Lustre: Skipped 1 previous similar message [ 8235.231101] Lustre: lustre-MDT0001: new disk, initializing [ 8235.429957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8235.452348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8240.489867] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8250.935311] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8256.878919] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8264.136834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.718330] Lustre: 213959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.952358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8266.129992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8272.817566] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8284.166411] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8288.621663] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8288.935377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8294.766172] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8295.202783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8302.051208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8302.063036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.096612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.123085] Lustre: Skipped 6 previous similar messages [ 8302.141777] Lustre: Skipped 8 previous similar messages [ 8312.292012] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8312.650632] Lustre: server umount lustre-OST0000 complete [ 8323.403220] Lustre: server umount lustre-MDT0000 complete [ 8327.136521] LustreError: 211981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531151 with bad export cookie 6245569402731759555 [ 8327.150333] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8327.155469] LustreError: 211981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8327.705432] Lustre: server umount lustre-MDT0001 complete [ 8331.418159] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:12:34 (1788531154) [ 8350.032359] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 8362.120744] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8401.466086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.648286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8401.677701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8401.681481] Lustre: Skipped 1 previous similar message [ 8401.734157] Lustre: lustre-MDT0000: new disk, initializing [ 8401.735551] Lustre: Skipped 1 previous similar message [ 8401.795357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8401.800887] Lustre: Skipped 1 previous similar message [ 8406.461287] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8420.122437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.235976] Lustre: 218574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8420.490246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8425.841920] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8436.584985] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8442.230470] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8449.496617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.755830] Lustre: 220546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8452.156490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8456.577235] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8466.810696] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.493213] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8471.714967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8476.304751] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8476.586208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8482.787863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.804162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.810810] Lustre: Skipped 3 previous similar messages [ 8482.817324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.817337] LustreError: Skipped 1 previous similar message [ 8482.854723] Lustre: Skipped 9 previous similar messages [ 8485.040897] Lustre: server umount lustre-OST0000 complete [ 8495.291226] Lustre: server umount lustre-MDT0000 complete [ 8498.160053] LustreError: 218587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.185856] LustreError: 218587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8499.329183] LustreError: 218566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531324 with bad export cookie 6245569402731760437 [ 8499.333389] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8499.339852] LustreError: 218566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8499.629177] Lustre: server umount lustre-MDT0001 complete [ 8509.175231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.741235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.748576] Lustre: Skipped 5 previous similar messages [ 8515.438495] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8524.725936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.813724] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8539.099986] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8545.632199] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8552.778224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.585832] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8568.502237] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.065279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8580.075875] LustreError: Skipped 1 previous similar message [ 8580.082596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.093482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8580.093490] Lustre: Skipped 3 previous similar messages [ 8580.129047] Lustre: Skipped 3 previous similar messages [ 8585.456929] Lustre: server umount lustre-OST0000 complete [ 8597.092950] Lustre: server umount lustre-MDT0000 complete [ 8601.633170] LustreError: 225639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531426 with bad export cookie 6245569402731761319 [ 8601.665532] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8602.167289] Lustre: server umount lustre-MDT0001 complete [ 8708.483734] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8711.783884] Key type lgssc unregistered [ 8712.283409] LNet: 226425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8712.298434] LNetError: 226425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8712.328845] LNet: Removed LNI 192.168.202.134@tcp [ 8713.504302] Key type .llcrypt unregistered [ 8713.507264] Key type ._llcrypt unregistered [ 8739.825336] Key type ._llcrypt registered [ 8739.827327] Key type .llcrypt registered [ 8739.934629] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 8757.766203] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8758.989673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8759.025111] alg: No test for adler32 (adler32-zlib) [ 8760.318757] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 8760.514461] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8762.176242] Key type lgssc registered [ 8763.615170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8803.418168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8803.446855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.798932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.831441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8804.908863] Lustre: lustre-MDT0000: new disk, initializing [ 8805.012684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.027898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8809.923907] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8822.413334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.519319] Lustre: 229777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8822.567563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8822.571290] Lustre: Skipped 1 previous similar message [ 8822.654399] Lustre: lustre-MDT0001: new disk, initializing [ 8822.738424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8822.768748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8822.781337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8827.829875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8837.138508] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8843.042149] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8849.949140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.212586] Lustre: lustre-OST0000: new disk, initializing [ 8850.221682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8850.231596] Lustre: 231743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8850.330986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8857.013335] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8858.225871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8858.240261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8858.360344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8867.050654] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8872.319436] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8872.530953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8877.514994] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8877.720711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8884.194188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.199186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8884.234600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.244795] Lustre: Skipped 1 previous similar message [ 8885.879624] Lustre: server umount lustre-OST0000 complete [ 8889.324509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8889.324900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.331520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.386549] Lustre: Skipped 2 previous similar messages [ 8894.439855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8894.446090] Lustre: Skipped 2 previous similar messages [ 8895.594816] Lustre: server umount lustre-MDT0000 complete [ 8899.014852] LustreError: 229768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531723 with bad export cookie 5230558741666184000 [ 8899.020807] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8899.023160] LustreError: 229768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8899.266810] Lustre: server umount lustre-MDT0001 complete [ 8917.937862] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8928.253820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.888400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.661873] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8934.375339] LustreError: 234677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8934.395949] LustreError: 234677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8938.472518] LustreError: 234676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8942.036140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8942.506734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8947.273569] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8950.515349] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8957.304846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8957.712092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.301518] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8972.392033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8972.547896] Lustre: lustre-OST0001: new disk, initializing [ 8972.551416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8972.556094] Lustre: 236983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8978.964963] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8979.572750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8979.584226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8979.719812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8991.137687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9001.170765] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9013.033856] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:23:55 (1788531835) [ 9015.785336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.807441] Lustre: Skipped 1 previous similar message [ 9015.813255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.827925] Lustre: Skipped 1 previous similar message [ 9020.901130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9020.911220] Lustre: Skipped 5 previous similar messages [ 9021.548293] Lustre: server umount lustre-MDT0000 complete [ 9024.904797] LustreError: 234657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788531849 with bad export cookie 5230558741666184882 [ 9024.909882] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.917237] LustreError: 234657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9025.238425] Lustre: server umount lustre-MDT0001 complete [ 9039.295898] Lustre: server umount lustre-OST0000 complete [ 9041.184119] Lustre: 227996:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788531850/real 1788531850] req@ffff8f13c6df2300 x1875411295383424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788531866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9041.213765] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.987435] Lustre: server umount lustre-OST0001 complete [ 9051.261937] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9062.936496] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9101.519247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.750374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9101.790259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9101.867721] Lustre: lustre-MDT0000: new disk, initializing [ 9101.956460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.962446] Lustre: Skipped 1 previous similar message [ 9101.976433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9108.349761] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9119.644988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.832637] Lustre: 240925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9119.980515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9119.998052] Lustre: Skipped 1 previous similar message [ 9120.175730] Lustre: lustre-MDT0001: new disk, initializing [ 9120.477201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9120.490774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9127.474225] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9137.632616] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9142.781524] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9148.680095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9149.025575] Lustre: lustre-OST0000: new disk, initializing [ 9149.030705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9149.036371] Lustre: 242895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9149.125248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.138555] Lustre: Skipped 1 previous similar message [ 9150.152107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9150.182151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9150.270430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9155.933933] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9165.820431] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9170.544871] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9170.807704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9175.720932] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9175.993856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9181.154453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9181.160389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9181.164468] LustreError: Skipped 1 previous similar message [ 9181.190663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9181.191896] Lustre: Skipped 1 previous similar message [ 9181.209170] Lustre: Skipped 1 previous similar message [ 9184.121846] Lustre: server umount lustre-OST0000 complete [ 9191.396713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.416451] Lustre: Skipped 1 previous similar message [ 9194.428187] Lustre: server umount lustre-MDT0000 complete [ 9196.522467] LustreError: 240932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.543555] LustreError: 240932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9198.386250] LustreError: 240919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532023 with bad export cookie 5230558741666187024 [ 9198.393256] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9198.404609] LustreError: 240919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9199.015330] Lustre: server umount lustre-MDT0001 complete [ 9208.627518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.396086] LustreError: 245112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.965912] LustreError: 245113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.056767] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9224.232957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.614363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9224.622996] Lustre: Skipped 1 previous similar message [ 9229.500700] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9236.847876] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9242.854599] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9250.296841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.350170] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9264.694613] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9411.560774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.580185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9411.594980] Lustre: Skipped 1 previous similar message [ 9411.597095] Lustre: Skipped 2 previous similar messages [ 9425.377338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9425.775586] Lustre: server umount lustre-OST0000 complete [ 9436.896982] Lustre: server umount lustre-MDT0000 complete [ 9437.166695] LustreError: 245109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9437.198171] LustreError: 245109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9441.295188] LustreError: 245094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532266 with bad export cookie 5230558741666187906 [ 9441.310570] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9441.320069] LustreError: 245094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9441.759974] Lustre: server umount lustre-MDT0001 complete [ 9534.570939] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 9537.231981] Key type lgssc unregistered [ 9537.473733] LNet: 248720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9537.478846] LNetError: 248720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9537.490393] LNet: Removed LNI 192.168.202.134@tcp [ 9538.120231] Key type .llcrypt unregistered [ 9538.122876] Key type ._llcrypt unregistered [ 9564.386899] Key type ._llcrypt registered [ 9564.390565] Key type .llcrypt registered [ 9564.505408] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9583.577638] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9584.531525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9584.645103] alg: No test for adler32 (adler32-zlib) [ 9585.663086] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 9585.976416] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 9587.640147] Key type lgssc registered [ 9588.583300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9637.060469] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9648.474764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9648.515092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.786495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9649.813540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9649.878169] Lustre: lustre-MDT0000: new disk, initializing [ 9649.929561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.951951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9654.857976] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9664.995526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.125359] Lustre: 252788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9665.188884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9665.193595] Lustre: Skipped 1 previous similar message [ 9665.262079] Lustre: lustre-MDT0001: new disk, initializing [ 9665.382973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.428661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9665.437823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9669.885803] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9674.392658] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9681.880385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.339911] Lustre: lustre-OST0000: new disk, initializing [ 9682.349846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9682.356870] Lustre: 254418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.491797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.158966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9688.170105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9688.285040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9689.349868] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9699.799404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9699.961547] Lustre: lustre-OST0001: new disk, initializing [ 9699.965080] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9699.969670] Lustre: 255292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9700.061872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9707.124544] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9709.183440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9709.203922] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9709.316477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9717.832047] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9727.240567] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9740.267331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.282444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9740.291208] Lustre: Skipped 1 previous similar message [ 9740.314994] Lustre: Skipped 3 previous similar messages [ 9742.450795] Lustre: server umount lustre-MDT0000 complete [ 9746.002865] LustreError: 253581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532570 with bad export cookie 748616565858549668 [ 9746.018611] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9746.025487] LustreError: 253581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9746.479083] Lustre: server umount lustre-MDT0001 complete [ 9760.851053] Lustre: server umount lustre-OST0000 complete [ 9774.912814] Lustre: server umount lustre-OST0001 complete [ 9788.738549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9798.456039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9807.877554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9817.917702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9837.284742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9847.940267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.965334] Lustre: lustre-MDT0000: reset Object Index mappings [ 9848.553182] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.578928] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9848.667977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9853.741100] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9853.931431] LustreError: 260301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.046775] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.557362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.567324] Lustre: lustre-MDT0001: reset Object Index mappings [ 9862.913577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9868.070789] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9871.327226] Lustre: 261444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9878.442966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.209838] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9888.057374] LustreError: 261801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.232273] LustreError: 261801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.256933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9893.265707] LustreError: 261801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9894.880865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.297322] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.312415] Lustre: Skipped 1 previous similar message [ 9900.531561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9901.778762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9910.077855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9913.717950] Lustre: 263316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9921.003652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.006080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.028693] Lustre: Skipped 5 previous similar messages [ 9921.038841] Lustre: Skipped 3 previous similar messages [ 9926.068758] Lustre: server umount lustre-MDT0000 complete [ 9926.116627] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.132028] LustreError: 260300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9930.490231] LustreError: 260282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532755 with bad export cookie 748616565858556458 [ 9930.503315] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9930.511602] LustreError: 260282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9930.828518] Lustre: server umount lustre-MDT0001 complete [ 9944.474778] Lustre: server umount lustre-OST0000 complete [ 9947.617592] Lustre: 250292:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788532756/real 1788532756] req@ffff8f13c8642d80 x1875412160474624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788532772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9947.637558] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.912364] Lustre: server umount lustre-OST0001 complete [ 9966.222402] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:39:49 (1788532789) [ 9982.381563] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9993.375269] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10031.828698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.995715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10032.015455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10032.081524] Lustre: lustre-MDT0000: new disk, initializing [10032.232590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10032.266572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10035.755044] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10045.806854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.014885] Lustre: 267570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10046.096208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10046.100955] Lustre: Skipped 1 previous similar message [10046.242622] Lustre: lustre-MDT0001: new disk, initializing [10046.489185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10046.510822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10050.371258] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10058.955688] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10065.014331] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.621961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.002214] Lustre: lustre-OST0000: new disk, initializing [10073.005332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10073.014275] Lustre: 269541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10074.332708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10074.351879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10074.503256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10080.527898] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10091.116246] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10096.941169] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10097.212875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10103.128866] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10103.411347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10105.328106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10105.334755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.345977] LustreError: Skipped 1 previous similar message [10105.358904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.442755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.446268] Lustre: Skipped 2 previous similar messages [10111.552786] Lustre: server umount lustre-OST0000 complete [10115.556316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.569211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10115.569466] Lustre: Skipped 2 previous similar messages [10115.583418] Lustre: Skipped 1 previous similar message [10125.793731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.804364] Lustre: Skipped 3 previous similar messages [10129.376860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10129.951411] Lustre: server umount lustre-MDT0000 complete [10130.918289] LustreError: 267579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.944515] LustreError: 267579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10133.767746] LustreError: 267562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788532958 with bad export cookie 748616565858558551 [10133.776832] LustreError: 267562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10133.784373] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10134.185463] Lustre: server umount lustre-MDT0001 complete [10155.639611] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10168.462621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.953151] LustreError: 272477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.992389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.000315] Lustre: Skipped 2 previous similar messages [10173.005682] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10181.813820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.953466] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10191.016359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10197.544299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.908432] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10215.222995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10215.632668] Lustre: lustre-OST0001: new disk, initializing [10215.642315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10215.650346] Lustre: 274788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10216.740880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10216.747739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10216.891790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10222.572287] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10233.006747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10237.056953] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10242.828883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10242.851053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10254.311913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10254.322920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.337342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10260.075364] Lustre: server umount lustre-MDT0000 complete [10263.011725] LustreError: 272472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.049032] LustreError: 272472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10264.478848] LustreError: 272457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533089 with bad export cookie 748616565858559433 [10264.486836] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10264.502586] LustreError: 272457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10264.929585] Lustre: server umount lustre-MDT0001 complete [10279.962717] Lustre: server umount lustre-OST0000 complete [10283.488263] Lustre: 250292:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788533092/real 1788533092] req@ffff8f14f61a5c00 x1875412160582656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788533108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10283.517492] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10283.537352] Lustre: Skipped 3 previous similar messages [10285.024391] Lustre: 250294:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788533093/real 1788533093] req@ffff8f13c3e21180 x1875412160582912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788533109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10285.409128] Lustre: server umount lustre-OST0001 complete [10303.394129] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10314.599434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.097074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10315.107578] Lustre: Skipped 3 previous similar messages [10320.046208] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10330.657585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.280803] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10339.647382] Lustre: 279270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10347.582880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10355.319100] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10359.297840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10365.836936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10371.585765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10374.625110] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10382.614881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10386.715271] Lustre: 281149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10392.678325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10392.764412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10407.485205] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:47:09 (1788533229) [10423.267238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10423.287974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10423.322407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10423.336139] Lustre: Skipped 4 previous similar messages [10429.447385] Lustre: server umount lustre-MDT0000 complete [10433.008026] LustreError: 278126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.020147] LustreError: 278126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10433.628540] LustreError: 278112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533258 with bad export cookie 748616565858565824 [10433.630603] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10433.638124] LustreError: 278112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10440.240656] Lustre: server umount lustre-MDT0001 complete [10450.647927] Lustre: server umount lustre-OST0000 complete [10455.277823] Lustre: server umount lustre-OST0001 complete [10464.303513] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [10477.684154] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10530.514914] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10544.277046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.635981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10544.670556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10544.803518] Lustre: lustre-MDT0000: new disk, initializing [10545.035117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10550.443441] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10563.177615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.396172] Lustre: 285683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10563.826174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10570.267721] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10584.502223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10584.931760] Lustre: 287319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10585.098227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10585.103261] Lustre: Skipped 5 previous similar messages [10586.552108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10592.591240] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10606.084449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.311792] Lustre: 288191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10607.695550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10607.708134] Lustre: Skipped 1 previous similar message [10607.883844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10613.896559] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10624.516703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10628.683921] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10628.688488] Lustre: Skipped 1 previous similar message [10649.859525] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:51:11 (1788533471) [10652.291112] Lustre: server umount lustre-MDT0000 complete [10654.192743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.215452] Lustre: Skipped 7 previous similar messages [10656.680379] LustreError: 288190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533481 with bad export cookie 748616565858573811 [10656.693139] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10656.694689] LustreError: 288190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10657.009079] Lustre: server umount lustre-MDT0001 complete [10672.386367] Lustre: server umount lustre-OST0000 complete [10676.139787] Lustre: 250292:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788533484/real 1788533484] req@ffff8f13c3d60700 x1875412160744448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788533500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10677.137897] Lustre: server umount lustre-OST0001 complete [10688.661091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.387171] LustreError: 290437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.404976] LustreError: 290437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10696.379598] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10706.549248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.487391] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10721.177899] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10729.098064] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10739.993670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10749.701188] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10759.201486] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10779.408328] loop1: detected capacity change from 0 to 204800000 [10779.854891] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10779.891250] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10779.922971] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10786.281551] loop1: detected capacity change from 0 to 204800000 [10793.418077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.611828] LustreError: 290427:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26058: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10793.622524] LustreError: 290427:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26058 [10793.627491] LustreError: 290427:0:(mgs_handler.c:517:mgs_target_reg()) fs26058-MDT0001: Error checking target index: rc = -13 [10793.636668] LustreError: fs26058-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10793.641350] LustreError: 294157:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10793.644958] LustreError: 294157:0:(tgt_mount.c:2132:server_put_super()) no obd fs26058-MDT0001 [10793.647854] LustreError: 294157:0:(tgt_mount.c:116:server_deregister_mount()) fs26058-MDT0001 not registered [10793.710912] Lustre: server umount fs26058-MDT0001 complete [10793.716323] LustreError: 294157:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10804.226033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.443674] Lustre: Setting parameter fs26058-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26058-MDT0001 [10804.536730] Lustre: srv-fs26058-MDT0001: No data found on store. Initialize space. [10804.542306] Lustre: Skipped 4 previous similar messages [10804.657947] Lustre: fs26058-MDT0001: new disk, initializing [10804.660245] Lustre: Skipped 3 previous similar messages [10804.754414] Lustre: srv-fs26058-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10804.764156] Lustre: 294623:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26058-MDT0001-mdtlov: can't create new llog: rc = -115 [10804.773231] LustreError: 294623:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10810.580269] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10818.226134] Lustre: server umount fs26058-MDT0001 complete [10831.354372] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10833.878451] loop1: detected capacity change from 0 to 204800000 [10836.469520] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10840.923290] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.417427] Lustre: 296192:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26058-MDT0001-mdtlov: can't create new llog: rc = -115 [10841.432934] LustreError: 296192:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10847.738454] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10854.654508] Lustre: server umount fs26058-MDT0001 complete [10865.637471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10865.644711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10865.654266] Lustre: Skipped 2 previous similar messages [10865.667253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10865.671342] Lustre: Skipped 6 previous similar messages [10871.987670] Lustre: server umount lustre-MDT0000 complete [10881.559334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10881.662711] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10887.155976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10887.750411] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10897.881197] loop1: detected capacity change from 0 to 204800000 [10904.156246] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.821033] Lustre: 298641:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26058-MDT0001-mdtlov: can't create new llog: rc = -115 [10904.833504] LustreError: 298641:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10910.942593] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10917.011210] Lustre: server umount fs26058-MDT0001 complete [10946.017670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10949.826454] Lustre: server umount lustre-OST0000 complete [10960.417085] Lustre: server umount lustre-MDT0000 complete [10964.831407] LustreError: 290419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788533789 with bad export cookie 748616565858576492 [10964.840787] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10964.848871] LustreError: 290419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10965.686045] Lustre: server umount lustre-MDT0001 complete [10980.253719] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [10983.499956] Key type lgssc unregistered [10983.920823] LNet: 300987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10983.928921] LNetError: 300987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10983.951507] LNet: Removed LNI 192.168.202.134@tcp [10984.786181] Key type .llcrypt unregistered [10984.790076] Key type ._llcrypt unregistered [11005.284463] Key type ._llcrypt registered [11005.286827] Key type .llcrypt registered [11005.374600] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11006.665191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11006.811830] alg: No test for adler32 (adler32-zlib) [11007.842716] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11008.091902] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11009.784232] Key type lgssc registered [11011.259098] Lustre: Echo OBD driver; http://www.lustre.org/ [11019.928851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11019.948833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11021.349602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11026.826923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11031.528388] LustreError: 302255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.560226] LustreError: 302255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11036.646931] LustreError: 302254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11037.546863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.975373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11042.629332] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11050.788510] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11058.937055] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11067.255981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.602163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.135809] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11083.007449] LustreError: 304084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.112431] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11088.363625] LustreError: 304706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.467374] LustreError: 304706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.501993] LustreError: 304706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11106.662921] LustreError: 304084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.703522] LustreError: 304084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11115.353358] loop1: detected capacity change from 0 to 204800000 [11115.615710] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11115.632827] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11115.648456] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11123.595334] loop1: detected capacity change from 0 to 204800000 [11127.142988] LustreError: 304085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.171430] LustreError: 304085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11131.394372] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.599277] Lustre: fs8173-OST0000: new disk, initializing [11131.613210] Lustre: srv-fs8173-OST0000: No data found on store. Initialize space. [11131.623104] Lustre: 306013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11131.775992] Lustre: fs8173-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11140.680818] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11150.930781] Lustre: server umount fs8173-OST0000 complete [11162.997542] LustreError: 304706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.023615] LustreError: 304706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11163.123690] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11166.837926] loop1: detected capacity change from 0 to 204800000 [11168.237235] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11173.459431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11173.795763] Lustre: fs8173-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11181.750352] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11191.738011] loop2: detected capacity change from 0 to 204800000 [11192.198119] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11192.218971] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11192.247645] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11199.189843] loop2: detected capacity change from 0 to 204800000 [11205.462535] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11205.548831] LustreError: fs8173-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11205.565321] LustreError: fs8173-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11205.583626] LustreError: 309057:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11205.602853] LustreError: 309057:0:(tgt_mount.c:2132:server_put_super()) no obd fs8173-OST0001 [11205.608822] LustreError: 309057:0:(tgt_mount.c:116:server_deregister_mount()) fs8173-OST0001 not registered [11205.635856] Lustre: server umount fs8173-OST0001 complete [11205.639383] LustreError: 309057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11219.354630] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11219.651302] Lustre: fs8173-OST0001: new disk, initializing [11219.656200] Lustre: srv-fs8173-OST0001: No data found on store. Initialize space. [11219.667756] Lustre: 309608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11219.804784] Lustre: fs8173-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11228.062294] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11229.534728] LustreError: 304085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.556111] LustreError: 304085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11239.230527] Lustre: server umount fs8173-OST0000 complete [11250.771783] Lustre: server umount fs8173-OST0001 complete [11265.373087] loop1: detected capacity change from 0 to 204800000 [11273.086960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11273.413116] Lustre: fs8173-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11281.562773] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11290.567358] loop2: detected capacity change from 0 to 204800000 [11298.664243] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11308.556970] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11316.714590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11319.736345] Lustre: server umount fs8173-OST0000 complete [11321.837563] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11330.652493] Lustre: server umount fs8173-OST0001 complete [11344.354557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11344.366841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.378225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11347.437913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11347.438276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11350.382685] Lustre: server umount lustre-OST0000 complete [11355.640714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11355.652161] LustreError: Skipped 1 previous similar message [11355.659275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11355.690071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11355.696646] Lustre: Skipped 1 previous similar message [11357.668157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11357.701972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11357.720551] Lustre: Skipped 1 previous similar message [11361.693286] Lustre: server umount lustre-MDT0000 complete [11362.796384] LustreError: 302249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11362.845995] LustreError: 302249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 70 previous similar messages [11366.158890] LustreError: 302235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534190 with bad export cookie 7053768844062902738 [11366.176615] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11366.559571] Lustre: server umount lustre-MDT0001 complete [11383.207205] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11386.672452] Key type lgssc unregistered [11386.980746] LNet: 315549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11386.986945] LNetError: 315549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.008246] LNet: Removed LNI 192.168.202.134@tcp [11388.170211] Key type .llcrypt unregistered [11388.173732] Key type ._llcrypt unregistered [11425.943470] Key type ._llcrypt registered [11425.945954] Key type .llcrypt registered [11426.028511] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [11444.717830] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11446.243062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11446.315765] alg: No test for adler32 (adler32-zlib) [11447.612752] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11448.036411] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11449.777083] Key type lgssc registered [11451.423887] Lustre: Echo OBD driver; http://www.lustre.org/ [11491.275863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11491.321183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.700631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11492.723608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11492.801602] Lustre: lustre-MDT0000: new disk, initializing [11492.893615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11492.908964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11498.096437] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11508.899488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.080691] Lustre: 319405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11509.145314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11509.149676] Lustre: Skipped 1 previous similar message [11509.262185] Lustre: lustre-MDT0001: new disk, initializing [11509.391634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11509.432244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11509.460531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11514.907201] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11525.621542] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11532.399535] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11539.918952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11540.280994] Lustre: lustre-OST0000: new disk, initializing [11540.292785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11540.300503] Lustre: 321377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11540.394489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11546.201043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11546.207396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11546.339029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11547.780406] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11558.745932] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11561.011612] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2 [11562.044807] Lustre: Unmounted lustre-client [11571.172697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11571.207748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11571.223361] Lustre: Skipped 1 previous similar message [11576.298967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11576.310216] Lustre: Skipped 1 previous similar message [11577.407775] Lustre: server umount lustre-OST0000 complete [11586.531310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.533144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.550340] Lustre: Skipped 2 previous similar messages [11586.558544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11587.935579] Lustre: server umount lustre-MDT0000 complete [11591.653243] LustreError: 319398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534416 with bad export cookie 16211864020662733639 [11591.654153] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11591.664233] LustreError: 319398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11592.067030] Lustre: server umount lustre-MDT0001 complete [11602.048403] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11604.667124] Key type lgssc unregistered [11617.056650] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [11617.316367] LNet: 323537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11617.345301] LNetError: 323537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11691.131662] LNet: Removed LNI 192.168.202.134@tcp [11692.170252] Key type .llcrypt unregistered [11692.173052] Key type ._llcrypt unregistered [11710.384590] Key type ._llcrypt registered [11710.389738] Key type .llcrypt registered [11710.512884] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [11725.867283] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11726.981748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11727.105651] alg: No test for adler32 (adler32-zlib) [11728.415153] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11728.607572] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11730.254633] Key type lgssc registered [11731.389792] Lustre: Echo OBD driver; http://www.lustre.org/ [11761.597559] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:09:44 (1788534584) [11769.231190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11769.299918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11770.575537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11770.601686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11770.665436] Lustre: lustre-MDT0000: new disk, initializing [11770.749056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11770.762123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11776.058218] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11787.364945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.563953] Lustre: 326755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11787.670495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11787.674181] Lustre: Skipped 1 previous similar message [11787.737204] Lustre: lustre-MDT0001: new disk, initializing [11787.828564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11787.871869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11787.889063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11792.932852] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11802.129521] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11807.774691] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11814.311929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11814.693219] Lustre: lustre-OST0000: new disk, initializing [11814.699610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11814.706682] Lustre: 328724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11814.796526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11821.205831] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11824.210614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11824.222324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11824.407403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11831.094487] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11845.094358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11845.100521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11845.106938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11845.122609] Lustre: Skipped 1 previous similar message [11850.214908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11850.223745] Lustre: Skipped 2 previous similar messages [11851.377844] Lustre: server umount lustre-OST0000 complete [11859.429651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.440115] LustreError: Skipped 1 previous similar message [11859.451894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.466835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11860.457648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11862.470576] Lustre: server umount lustre-MDT0000 complete [11866.928725] LustreError: 326745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534691 with bad export cookie 8454010178625815517 [11866.958167] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11866.960706] LustreError: 326745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11867.380920] Lustre: server umount lustre-MDT0001 complete [11880.171620] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11882.985306] Key type lgssc unregistered [11883.401874] LNet: 330631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11883.409387] LNetError: 330631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11883.440264] LNet: Removed LNI 192.168.202.134@tcp [11884.392845] Key type .llcrypt unregistered [11884.394899] Key type ._llcrypt unregistered [11906.804877] Key type ._llcrypt registered [11906.807360] Key type .llcrypt registered [11906.921575] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11908.477216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11908.485722] alg: No test for adler32 (adler32-zlib) [11909.609802] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [11909.937308] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11911.664438] Key type lgssc registered [11913.098255] Lustre: Echo OBD driver; http://www.lustre.org/ [11921.713517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11921.761984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11923.550648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11929.109485] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11933.673970] LustreError: 331903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.694569] LustreError: 331903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11938.278576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.722838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11944.870870] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11953.541343] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11962.123811] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.064256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.415673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11975.511827] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11983.847565] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11996.278765] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11998.114695] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12004.322417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12004.330857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12004.356289] Lustre: Skipped 1 previous similar message [12004.367066] Lustre: Skipped 2 previous similar messages [12007.301156] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [12008.571560] Lustre: server umount lustre-MDT0000 complete [12009.448405] LustreError: 331900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.480098] LustreError: 331900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12012.380815] LustreError: 333002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12013.322083] LustreError: 331884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534838 with bad export cookie 17822448580623900077 [12013.330390] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12013.346900] LustreError: 331884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12014.565249] LustreError: 331899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.565358] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.582673] Lustre: Skipped 1 previous similar message [12014.591930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12017.528693] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [12018.595454] LustreError: 331898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.618091] LustreError: 331898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12019.917227] Lustre: server umount lustre-MDT0001 complete [12024.015907] Lustre: server umount lustre-OST0000 complete [12032.869130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12038.903540] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12042.204484] Lustre: 336085:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.134@tcp [12044.574227] Lustre: server umount MGS complete [12055.746049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.273708] LustreError: 336658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.367833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12062.138106] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12073.797467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.108486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12079.868320] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12089.571264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12090.029400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12097.419503] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12106.623233] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12108.587697] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12116.033724] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12118.029558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12124.131468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12124.138773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.143601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.151468] Lustre: Skipped 1 previous similar message [12124.164902] Lustre: Skipped 2 previous similar messages [12130.386627] Lustre: server umount lustre-OST0000 complete [12135.402555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12135.416574] LustreError: Skipped 1 previous similar message [12135.418395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.454692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.465862] Lustre: Skipped 2 previous similar messages [12141.553573] Lustre: server umount lustre-MDT0000 complete [12142.582334] LustreError: 337386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.607599] LustreError: 337386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12146.549821] LustreError: 336638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788534971 with bad export cookie 17822448580623900917 [12146.550777] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12146.562217] LustreError: 336638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12147.084327] Lustre: server umount lustre-MDT0001 complete [12161.796215] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12165.587230] Key type lgssc unregistered [12166.143708] LNet: 340099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12166.158593] LNetError: 340099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12166.180075] LNet: Removed LNI 192.168.202.134@tcp [12167.139886] Key type .llcrypt unregistered [12167.143336] Key type ._llcrypt unregistered [12190.411310] Key type ._llcrypt registered [12190.414440] Key type .llcrypt registered [12190.634830] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [12208.711994] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [12210.054917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12210.153942] alg: No test for adler32 (adler32-zlib) [12211.515198] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [12211.778518] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [12213.472485] Key type lgssc registered [12214.973563] Lustre: Echo OBD driver; http://www.lustre.org/ [12257.309567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12257.340699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.654041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12258.692513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12258.856207] Lustre: lustre-MDT0000: new disk, initializing [12258.957238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12259.016554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12264.894341] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12279.049537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12279.300526] Lustre: 343259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12279.412496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12279.415096] Lustre: Skipped 1 previous similar message [12279.464232] Lustre: lustre-MDT0001: new disk, initializing [12279.529800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12279.571904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12279.589325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12285.383203] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12297.766522] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12304.683109] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12312.378587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.807602] Lustre: lustre-OST0000: new disk, initializing [12312.819713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12312.833122] Lustre: 345228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12312.959083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12321.543277] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12322.939048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12322.952687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12323.142625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12333.162576] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12338.948166] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12339.266978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12345.279920] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12345.593684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12348.897220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12348.907516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.920579] LustreError: Skipped 1 previous similar message [12348.950475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12348.964805] Lustre: Skipped 1 previous similar message [12354.020823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12354.025839] Lustre: Skipped 1 previous similar message [12354.305802] Lustre: server umount lustre-OST0000 complete [12359.144586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.155076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12359.168609] Lustre: Skipped 2 previous similar messages [12359.190882] Lustre: Skipped 1 previous similar message [12364.258564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.266418] Lustre: Skipped 1 previous similar message [12365.645322] Lustre: server umount lustre-MDT0000 complete [12369.677920] LustreError: 343251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535194 with bad export cookie 8399597905557749455 [12369.680782] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12369.691599] LustreError: 343251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12370.364185] Lustre: server umount lustre-MDT0001 complete [12381.345882] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:20:03 (1788535203) [12399.441668] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [12411.347601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [12449.670966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.929967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12449.983481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12450.069722] Lustre: lustre-MDT0000: new disk, initializing [12450.160444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12450.174279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12455.340512] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12467.427556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.669500] Lustre: 349989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12467.884221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12467.914189] Lustre: Skipped 1 previous similar message [12468.045507] Lustre: lustre-MDT0001: new disk, initializing [12468.179326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12468.212977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12468.244130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12473.278438] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12484.707799] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12491.610356] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12499.181181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12499.370466] Lustre: lustre-OST0000: new disk, initializing [12499.373585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12499.377841] Lustre: 351960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12499.425146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12505.926312] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12509.775438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12509.792275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12509.978643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12518.655130] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12561.380828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12561.387554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.390611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.412246] Lustre: Skipped 1 previous similar message [12564.976585] Lustre: server umount lustre-OST0000 complete [12570.593738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12570.596807] LustreError: Skipped 1 previous similar message [12570.612721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12570.630764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.637057] Lustre: Skipped 1 previous similar message [12575.309123] Lustre: server umount lustre-MDT0000 complete [12576.740063] LustreError: 350002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.772217] LustreError: 350002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12579.797759] LustreError: 349983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788535404 with bad export cookie 8399597905557750330 [12579.805114] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12579.806872] LustreError: 349983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12580.318674] Lustre: server umount lustre-MDT0001 complete [12593.606751] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12597.337350] Key type lgssc unregistered [12597.762356] LNet: 354133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12597.768535] LNetError: 354133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12597.793922] LNet: Removed LNI 192.168.202.134@tcp [12598.836620] Key type .llcrypt unregistered [12598.839695] Key type ._llcrypt unregistered [12694.500184] Key type ._llcrypt registered [12694.503291] Key type .llcrypt registered [12694.654738] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12694.904386] Key type .llcrypt unregistered [12694.906140] Key type ._llcrypt unregistered [12772.673296] Key type ._llcrypt registered [12772.675930] Key type .llcrypt registered [12772.811163] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12773.211568] Key type .llcrypt unregistered [12773.214963] Key type ._llcrypt unregistered [12805.913780] Key type ._llcrypt registered [12805.915447] Key type .llcrypt registered [12806.116729] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12806.698710] Key type .llcrypt unregistered [12806.702722] Key type ._llcrypt unregistered [12842.800109] Key type ._llcrypt registered [12842.802544] Key type .llcrypt registered [12842.923250] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12843.296987] Key type .llcrypt unregistered [12843.299351] Key type ._llcrypt unregistered [12934.575566] Key type ._llcrypt registered [12934.578328] Key type .llcrypt registered [12934.692796] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12934.999550] Key type .llcrypt unregistered [12935.000992] Key type ._llcrypt unregistered [12970.911686] Key type ._llcrypt registered [12970.915856] Key type .llcrypt registered [12971.085425] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12971.465542] Key type .llcrypt unregistered [12971.468178] Key type ._llcrypt unregistered [13008.624217] Key type ._llcrypt registered [13008.628470] Key type .llcrypt registered [13008.786510] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13009.467282] Key type .llcrypt unregistered [13009.469085] Key type ._llcrypt unregistered [13043.361718] Key type ._llcrypt registered [13043.364452] Key type .llcrypt registered [13043.491394] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13043.787524] Key type .llcrypt unregistered [13043.790228] Key type ._llcrypt unregistered [13084.401378] Key type ._llcrypt registered [13084.407105] Key type .llcrypt registered [13084.647471] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13084.939605] Key type .llcrypt unregistered [13084.943033] Key type ._llcrypt unregistered [13120.511736] Key type ._llcrypt registered [13120.514437] Key type .llcrypt registered [13120.601192] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13121.046394] Key type .llcrypt unregistered [13121.050401] Key type ._llcrypt unregistered [13191.067419] Key type ._llcrypt registered [13191.069343] Key type .llcrypt registered [13191.165578] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13191.442450] Key type .llcrypt unregistered [13191.445692] Key type ._llcrypt unregistered [13225.264552] Key type ._llcrypt registered [13225.268638] Key type .llcrypt registered [13225.392465] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13225.689811] Key type .llcrypt unregistered [13225.708406] Key type ._llcrypt unregistered [13257.061521] Key type ._llcrypt registered [13257.062921] Key type .llcrypt registered [13257.154482] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13257.399658] Key type .llcrypt unregistered [13257.403394] Key type ._llcrypt unregistered [13317.559549] Key type ._llcrypt registered [13317.564354] Key type .llcrypt registered [13317.712025] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [13334.980345] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13336.180925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13336.381540] alg: No test for adler32 (adler32-zlib) [13337.477316] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [13337.707973] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13339.408214] Key type lgssc registered [13340.897380] Lustre: Echo OBD driver; http://www.lustre.org/ [13378.293634] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13378.324456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.653284] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13379.708757] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13379.829761] Lustre: tst250fs-MDT0000: new disk, initializing [13379.923895] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13379.943793] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13384.337440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13393.917364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13394.070554] Lustre: 374417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13394.194387] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13394.201482] Lustre: Skipped 1 previous similar message [13394.271448] Lustre: tst250fs-MDT0001: new disk, initializing [13394.383796] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13394.424892] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13394.434805] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13399.144565] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13404.418882] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13411.588418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13411.964460] Lustre: tst250fs-OST0000: new disk, initializing [13411.981066] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13411.994198] Lustre: 376051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13412.152686] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13416.656168] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13416.675867] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13416.842247] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13419.424635] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13429.615714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13429.763756] Lustre: tst250fs-OST0001: new disk, initializing [13429.766685] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13429.771248] Lustre: 376922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13429.875171] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13437.135556] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13438.590718] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13438.601530] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13438.728562] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13442.543076] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13445.170077] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13445.175512] Lustre: Skipped 2 previous similar messages [13469.670226] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13469.688496] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13469.690799] Lustre: Skipped 1 previous similar message [13469.707215] Lustre: Skipped 3 previous similar messages [13473.469179] Lustre: server umount tst250fs-MDT0000 complete [13474.790064] LustreError: 374429:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13474.810972] LustreError: 374429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13478.079420] LustreError: 374408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788536302 with bad export cookie 8388058663876351082 [13478.084191] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13478.111070] LustreError: 374408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13478.677106] Lustre: server umount tst250fs-MDT0001 complete [13492.520747] Lustre: server umount tst250fs-OST0000 complete [13495.265738] Lustre: 372641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536304/real 1788536304] req@ffff8f14d06a5c00 x1875416094676992/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788536320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13495.292330] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13495.308570] Lustre: Skipped 2 previous similar messages [13497.140275] Lustre: server umount tst250fs-OST0001 complete [13517.695717] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [13528.100381] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13567.320494] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:39:49 (1788536389) [13568.734486] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13584.951133] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [13598.684629] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13629.445191] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13306 sec ========= 11:40:52 (1788536452) [13631.092216] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:40:53 (1788536453) === [13634.643634] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:40:57 (1788536457) === [13675.095858] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13678.110160] Key type lgssc unregistered [13678.390669] LNet: 386067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13678.395380] LNetError: 386067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13678.405849] LNet: Removed LNI 192.168.202.134@tcp [13679.076295] Key type .llcrypt unregistered [13679.081707] Key type ._llcrypt unregistered