[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454266101 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002322] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006732] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007037] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.010103] LSM: Security Framework initializing [ 0.011081] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013069] *** VALIDATE selinux *** [ 0.021796] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026142] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029142] *** VALIDATE tmpfs *** [ 0.031284] *** VALIDATE proc *** [ 0.032296] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039000] Speculative Store Bypass: Vulnerable [ 0.040804] debug: unmapping init [mem 0xffffffffa6e59000-0xffffffffa6e60fff] [ 0.042177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043757] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051350] rcu: Hierarchical SRCU implementation. [ 0.053523] smp: Bringing up secondary CPUs ... [ 0.054597] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.058309] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137000] node 0 deferred pages initialised in 74ms [ 0.141102] devtmpfs: initialized [ 0.142334] x86/mm: Memory block size: 128MB [ 0.145971] gcov: version magic: 0x41383552 [ 0.149310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155346] pinctrl core: initialized pinctrl subsystem [ 0.157197] [ 0.157799] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162013] ** ** [ 0.165014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167012] ** ** [ 0.169016] ** This means that this kernel is built to expose internal ** [ 0.172014] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.177018] ** ** [ 0.179013] ** If you see this message and you are not debugging the ** [ 0.182014] ** kernel, report this immediately to your vendor! ** [ 0.184013] ** ** [ 0.186015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189017] ************************************************************* [ 0.191756] NET: Registered protocol family 16 [ 0.192437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201106] cpuidle: using governor menu [ 0.202829] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205475] PCI: Using configuration type 1 for base access [ 0.207152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216075] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220086] cryptd: max_cpu_qlen set to 1000 [ 0.223635] ACPI: Added _OSI(Module Device) [ 0.225013] ACPI: Added _OSI(Processor Device) [ 0.226009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230119] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235596] ACPI: Interpreter enabled [ 0.236056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238014] ACPI: Using IOAPIC for interrupt routing [ 0.240118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254735] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268550] acpiphp: Slot [2] registered [ 0.270132] acpiphp: Slot [5] registered [ 0.272188] acpiphp: Slot [6] registered [ 0.273094] acpiphp: Slot [7] registered [ 0.274072] acpiphp: Slot [8] registered [ 0.275052] acpiphp: Slot [9] registered [ 0.276127] acpiphp: Slot [10] registered [ 0.278176] acpiphp: Slot [3] registered [ 0.279104] acpiphp: Slot [4] registered [ 0.281102] acpiphp: Slot [11] registered [ 0.282143] acpiphp: Slot [12] registered [ 0.283080] acpiphp: Slot [13] registered [ 0.285105] acpiphp: Slot [14] registered [ 0.286069] acpiphp: Slot [15] registered [ 0.287010] acpiphp: Slot [16] registered [ 0.287973] acpiphp: Slot [17] registered [ 0.289115] acpiphp: Slot [18] registered [ 0.291122] acpiphp: Slot [19] registered [ 0.292098] acpiphp: Slot [20] registered [ 0.294131] acpiphp: Slot [21] registered [ 0.295127] acpiphp: Slot [22] registered [ 0.297101] acpiphp: Slot [23] registered [ 0.299113] acpiphp: Slot [24] registered [ 0.300106] acpiphp: Slot [25] registered [ 0.302127] acpiphp: Slot [26] registered [ 0.304121] acpiphp: Slot [27] registered [ 0.305086] acpiphp: Slot [28] registered [ 0.306081] acpiphp: Slot [29] registered [ 0.307069] acpiphp: Slot [30] registered [ 0.308071] acpiphp: Slot [31] registered [ 0.309074] PCI host bridge to bus 0000:00 [ 0.311023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328780] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332353] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352424] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354586] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359832] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.378017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.382014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388220] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.482161] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533282] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.563024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.617647] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.627023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.637018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677444] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.694015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.702016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.730023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740041] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.775017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.785790] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788290] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789214] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.790343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.798031] iommu: Default domain type: Passthrough [ 0.799325] SCSI subsystem initialized [ 0.800058] ACPI: bus type USB registered [ 0.800837] usbcore: registered new interface driver usbfs [ 0.802036] usbcore: registered new interface driver hub [ 0.803067] usbcore: registered new device driver usb [ 0.805205] pps_core: LinuxPPS API ver. 1 registered [ 0.807016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.810062] PTP clock support registered [ 0.813072] EDAC MC: Ver: 3.0.0 [ 0.815180] PCI: Using ACPI for IRQ routing [ 0.818391] NetLabel: Initializing [ 0.819007] NetLabel: domain hash size = 128 [ 0.820006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.821076] NetLabel: unlabeled traffic allowed by default [ 0.822102] vgaarb: loaded [ 0.824218] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.825011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.830394] clocksource: Switched to clocksource kvm-clock [ 0.933303] VFS: Disk quotas dquot_6.6.0 [ 0.934760] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.936820] *** VALIDATE ramfs *** [ 0.937817] *** VALIDATE hugetlbfs *** [ 0.939021] pnp: PnP ACPI init [ 0.940963] pnp: PnP ACPI: found 6 devices [ 0.958306] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.961084] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.962791] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.964523] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966391] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968487] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.970988] NET: Registered protocol family 2 [ 0.972810] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.977482] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.980462] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.984952] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.988522] TCP: Hash tables configured (established 65536 bind 65536) [ 0.990807] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.993287] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.995553] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.998070] NET: Registered protocol family 1 [ 1.000248] RPC: Registered named UNIX socket transport module. [ 1.002106] RPC: Registered udp transport module. [ 1.003388] RPC: Registered tcp transport module. [ 1.004633] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.006453] NET: Registered protocol family 44 [ 1.007684] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.009653] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.011308] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.013382] PCI: CLS 0 bytes, default 64 [ 1.014781] Unpacking initramfs... [ 2.444630] debug: unmapping init [mem 0xffff8aecfcc54000-0xffff8aecfffbffff] [ 2.451280] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.452886] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.456606] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.934781] Initialise system trusted keyrings [ 2.936215] Key type blacklist registered [ 2.937690] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.948417] zbud: loaded [ 2.952902] *** VALIDATE nfs *** [ 2.954357] *** VALIDATE nfs4 *** [ 2.955990] pstore: using deflate compression [ 2.959045] Platform Keyring initialized [ 3.048292] NET: Registered protocol family 38 [ 3.049480] Key type asymmetric registered [ 3.050374] Asymmetric key parser 'x509' registered [ 3.051478] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.053643] io scheduler mq-deadline registered [ 3.055265] io scheduler kyber registered [ 3.056664] io scheduler bfq registered [ 3.058343] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.061149] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.063684] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.066689] ACPI: Power Button [PWRF] [ 3.070867] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.075681] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.094508] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.105177] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.117791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.144983] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.172868] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.177167] Non-volatile memory driver v1.3 [ 3.178441] Linux agpgart interface v0.103 [ 3.202487] virtio_blk virtio1: [vda] 146608 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.204175] vda: detected capacity change from 0 to 75063296 [ 3.214803] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.217869] vdb: detected capacity change from 0 to 1073741824 [ 3.238367] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.240175] vdc: detected capacity change from 0 to 2621440000 [ 3.251572] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.254192] vdd: detected capacity change from 0 to 2621440000 [ 3.267430] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269835] vde: detected capacity change from 0 to 4294967296 [ 3.282339] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.284647] vdf: detected capacity change from 0 to 4294967296 [ 3.289684] libphy: Fixed MDIO Bus: probed [ 3.293764] usbcore: registered new interface driver usbserial_generic [ 3.295152] usbserial: USB Serial support registered for generic [ 3.296410] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.299088] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.300203] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.301724] mousedev: PS/2 mouse device common for all mice [ 3.303923] rtc_cmos 00:05: RTC can wake from S4 [ 3.305889] rtc_cmos 00:05: registered as rtc0 [ 3.306093] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.307756] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.312613] intel_pstate: CPU model not supported [ 3.314913] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317213] hid: raw HID events driver (C) Jiri Kosina [ 3.319104] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.319807] usbcore: registered new interface driver usbhid [ 3.323399] usbhid: USB HID core driver [ 3.324309] drop_monitor: Initializing network drop monitor service [ 3.325647] Initializing XFRM netlink socket [ 3.326762] NET: Registered protocol family 10 [ 3.328965] Segment Routing with IPv6 [ 3.329929] NET: Registered protocol family 17 [ 3.331305] mpls_gso: MPLS GSO support [ 3.335386] RAS: Correctable Errors collector initialized. [ 3.336817] AVX version of gcm_enc/dec engaged. [ 3.337680] AES CTR mode by8 optimization enabled [ 3.399604] sched_clock: Marking stable (3399572807, 0)->(4267128214, -867555407) [ 3.402134] registered taskstats version 1 [ 3.403626] Loading compiled-in X.509 certificates [ 3.404956] zswap: loaded using pool lzo/zbud [ 3.424411] Key type big_key registered [ 3.435632] Key type encrypted registered [ 3.437305] ima: No TPM chip found, activating TPM-bypass! [ 3.439349] ima: Allocated hash algorithm: sha1 [ 3.440800] ima: No architecture policies found [ 3.442581] evm: Initialising EVM extended attributes: [ 3.444364] evm: security.selinux [ 3.445539] evm: security.ima [ 3.446657] evm: security.capability [ 3.448027] evm: HMAC attrs: 0x1 [ 3.450327] rtc_cmos 00:05: setting system clock to 2026-08-27 00:33:26 UTC (1787790806) [ 3.456084] debug: unmapping init [mem 0xffffffffa7e03000-0xffffffffa7ffffff] [ 3.459047] debug: unmapping init [mem 0xffffffffa6b82000-0xffffffffa6e58fff] [ 3.468079] Write protecting the kernel read-only data: 28672k [ 3.471512] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 3.474351] debug: unmapping init [mem 0xffffffffa5b14000-0xffffffffa5bfffff] [ 3.508722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.514697] systemd[1]: Detected virtualization kvm. [ 3.516170] systemd[1]: Detected architecture x86-64. [ 3.517286] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.544296] systemd[1]: No hostname configured. [ 3.546081] systemd[1]: Set hostname to . [ 3.547770] random: systemd: uninitialized urandom read (16 bytes read) [ 3.549282] systemd[1]: Initializing machine ID from random generator. [ 3.667359] random: systemd: uninitialized urandom read (16 bytes read) [ 3.670038] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.673875] random: systemd: uninitialized urandom read (16 bytes read) [ 3.676474] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.680817] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.261965] device-mapper: uevent: version 1.0.3 [ 4.264686] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.951815] virtio_net virtio0 ens2: renamed from eth0 [ 4.972762] random: fast init done [ 5.012576] scsi host0: ata_piix [ 5.021717] scsi host1: ata_piix [ 5.023348] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.026284] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.177631] dracut-initqueue[598]: RTNETLINK answers: File exists [ 9.952835] random: crng init done [ 9.953740] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.211615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.261782] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.475357] SELinux: Disabled at runtime. [ 11.545560] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.551678] systemd[1]: Detected virtualization kvm. [ 11.553040] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.027110] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.031592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.036949] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.040673] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.044449] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.051125] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.055970] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ 12.197811] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.507276] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.771744] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.776801] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.998767] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.008087] EDAC sbridge: Ver: 1.1.2 [ 14.612390] Key type dns_resolver registered [ 14.908638] NFS: Registering the id_resolver key type [ 14.911888] Key type id_resolver registered [ 14.913164] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-server login: [ 42.761807] libcfs: loading out-of-tree module taints kernel. [ 42.780048] Key type ._llcrypt registered [ 42.781482] Key type .llcrypt registered [ 42.845577] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 52.758010] hrtimer: interrupt took 7182800 ns [ 54.497318] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 55.745420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.756395] alg: No test for adler32 (adler32-zlib) [ 57.135660] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 57.771694] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 59.511153] Key type lgssc registered [ 60.863550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.571828] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.080430] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 123.551640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 123.576928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 124.815925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.839544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.902979] Lustre: lustre-MDT0000: new disk, initializing [ 124.943556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.959893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 128.396442] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 139.544124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.620252] Lustre: 6496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 139.648989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 139.654141] Lustre: Skipped 1 previous similar message [ 139.754699] Lustre: lustre-MDT0001: new disk, initializing [ 139.829823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.861218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 139.870832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 144.033421] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.123755] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 157.921634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 158.121988] Lustre: lustre-OST0000: new disk, initializing [ 158.123831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 158.126621] Lustre: 8434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 158.174838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.232709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.236943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.310815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 163.802906] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.405222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.487779] Lustre: lustre-OST0001: new disk, initializing [ 177.490649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 177.494628] Lustre: 9505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.548372] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.112360] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.870892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 183.880105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 183.967783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 193.970466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.028852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 207.793149] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing check_logdir /tmp/testlogs/ [ 212.234188] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing yml_node [ 215.298514] Lustre: DEBUG MARKER: Client: 2.17.57.50 [ 217.557914] Lustre: DEBUG MARKER: MDS: 2.17.57.50 [ 219.705884] Lustre: DEBUG MARKER: OSS: 2.17.57.50 [ 220.990868] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 20:37:03 EDT 2026 [ 234.447597] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 235.674430] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 252.385240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 252.400514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.418824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 255.468026] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.479693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 255.488161] Lustre: Skipped 1 previous similar message [ 255.510073] Lustre: Skipped 3 previous similar messages [ 258.058980] Lustre: server umount lustre-MDT0000 complete [ 261.447890] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791064 with bad export cookie 15979193883581207044 [ 261.451291] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 261.462898] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 265.696343] LustreError: 6504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.697345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.716229] LustreError: 6504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 265.745825] Lustre: Skipped 1 previous similar message [ 265.748913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.757982] Lustre: Skipped 1 previous similar message [ 266.720402] LustreError: 8132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.746935] LustreError: 8132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 267.829431] Lustre: server umount lustre-MDT0001 complete [ 272.120898] Lustre: server umount lustre-OST0000 complete [ 276.540545] Lustre: server umount lustre-OST0001 complete [ 283.791637] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 292.065801] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 326.075799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.302363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 326.332713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 326.394443] Lustre: lustre-MDT0000: new disk, initializing [ 326.456213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.467911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 331.297541] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 341.166632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.272599] Lustre: 15054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 341.321258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 341.324825] Lustre: Skipped 1 previous similar message [ 341.387178] Lustre: lustre-MDT0001: new disk, initializing [ 341.499742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 341.547631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 341.573264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 346.316499] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 355.687989] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 360.862669] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 367.029188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.375331] Lustre: lustre-OST0000: new disk, initializing [ 367.383805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 367.391845] Lustre: 17026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 367.487502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.803672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 368.812059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 368.963131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 373.716961] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 382.024203] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 386.722922] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 386.895994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 391.910933] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 392.092580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 394.721341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 394.730575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.741523] LustreError: Skipped 1 previous similar message [ 394.743465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 394.753091] Lustre: Skipped 2 previous similar messages [ 394.774204] Lustre: Skipped 3 previous similar messages [ 399.861951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 399.876572] Lustre: Skipped 1 previous similar message [ 400.390052] Lustre: server umount lustre-OST0000 complete [ 404.961486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.980995] Lustre: Skipped 1 previous similar message [ 410.081666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 410.087594] Lustre: Skipped 3 previous similar messages [ 418.271196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 418.561790] Lustre: server umount lustre-MDT0000 complete [ 420.321019] LustreError: 15065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.343041] LustreError: 15065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 422.054748] LustreError: 15046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791225 with bad export cookie 15979193883581208577 [ 422.068368] LustreError: 15046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 422.071516] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.432956] Lustre: server umount lustre-MDT0001 complete [ 428.726332] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:40:30 (1787791230) [ 445.413463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.858263] LustreError: 19902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.925271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.892414] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 451.048168] LustreError: 19903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.409149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.290398] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 469.629610] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 474.916785] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 476.719665] Lustre: Failing over lustre-MDT0000 [ 476.933309] Lustre: server umount lustre-MDT0000 complete [ 478.175892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 478.181325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.188204] LustreError: 19903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.198758] LustreError: 19903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 494.766060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.877122] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.447513] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 500.194228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 500.200839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 500.216537] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 507.762836] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 515.553927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.555083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 515.556600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.556607] Lustre: Skipped 2 previous similar messages [ 515.564748] Lustre: Skipped 2 previous similar messages [ 517.594243] Lustre: server umount lustre-MDT0000 complete [ 520.673422] LustreError: 19897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.696774] LustreError: 19897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 520.970188] LustreError: 19882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791324 with bad export cookie 15979193883581209991 [ 520.980676] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.981031] LustreError: 19882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 521.193895] Lustre: server umount lustre-MDT0001 complete [ 528.912898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.391622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.399905] Lustre: Skipped 2 previous similar messages [ 533.863603] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 542.304566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.153858] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 552.866290] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.565868] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 559.004522] Lustre: Failing over lustre-MDT0000 [ 559.076688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.082806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.085176] Lustre: Skipped 1 previous similar message [ 559.091033] Lustre: Skipped 2 previous similar messages [ 559.198686] Lustre: server umount lustre-MDT0000 complete [ 564.195780] LustreError: 23240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.229885] LustreError: 23240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 577.956125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.057647] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.120993] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 583.650307] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 583.667252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 583.696158] Lustre: Skipped 1 previous similar message [ 583.722078] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 593.766269] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 599.009068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 603.761835] Lustre: server umount lustre-MDT0000 complete [ 606.916808] LustreError: 23223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791409 with bad export cookie 15979193883581210929 [ 606.930141] LustreError: 23223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 606.940161] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.266981] Lustre: server umount lustre-MDT0001 complete [ 614.596225] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:43:37 (1787791417) [ 628.731991] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 638.284686] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 676.402710] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 689.713369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.883041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 689.901586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 689.944186] Lustre: lustre-MDT0000: new disk, initializing [ 689.992589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.998723] Lustre: Skipped 2 previous similar messages [ 690.007432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 693.187418] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 703.121657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.303063] Lustre: 29613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 703.680115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 708.875172] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 717.596644] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 723.096576] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 729.536687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 729.903468] Lustre: 31584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 731.701479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 731.745855] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 731.839778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 731.857750] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 736.070477] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 745.145905] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 750.909447] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 773.091905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.098119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.107042] Lustre: Skipped 2 previous similar messages [ 773.130094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.135358] Lustre: Skipped 2 previous similar messages [ 775.353640] Lustre: server umount lustre-OST0000 complete [ 784.937670] Lustre: server umount lustre-MDT0000 complete [ 788.453454] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 788.466733] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 788.939040] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791591 with bad export cookie 15979193883581211335 [ 788.949043] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 788.949461] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.254929] Lustre: server umount lustre-MDT0001 complete [ 797.926859] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 800.548042] Key type lgssc unregistered [ 800.935933] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.940230] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.951706] LNet: Removed LNI 192.168.201.148@tcp [ 801.743181] Key type .llcrypt unregistered [ 801.744856] Key type ._llcrypt unregistered [ 824.152949] Key type ._llcrypt registered [ 824.155109] Key type .llcrypt registered [ 824.241511] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 825.369803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.459904] alg: No test for adler32 (adler32-zlib) [ 826.531359] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 826.740039] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 828.375968] Key type lgssc registered [ 829.613293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 837.120525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 837.145500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.714387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.669860] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 848.865609] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.885972] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 851.620886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.907622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 856.034693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 862.655848] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 868.153825] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.777410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.210550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.268896] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 880.502547] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 883.458731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 887.530831] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.693434] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:48:23 (1787791703) [ 902.177835] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 903.700513] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 909.027870] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:48:31 (1787791711) [ 916.822914] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:48:39 (1787791719) [ 917.639307] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 924.902563] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 924.908411] Lustre: Skipped 2 previous similar messages [ 926.436626] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:48:49 (1787791729) [ 929.790480] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 929.794211] Lustre: Skipped 3 previous similar messages [ 945.617275] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 945.621933] Lustre: Skipped 6 previous similar messages [ 948.007901] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:49:10 (1787791750) [ 950.439859] Lustre: *** cfs_fail_loc=131b, val=2*** [ 953.479519] Lustre: *** cfs_fail_loc=131b, val=2*** [ 962.190916] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:49:24 (1787791764) [ 964.039587] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 964.044878] Lustre: Skipped 3 previous similar messages [ 975.709590] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:49:38 (1787791778) [ 989.154691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.160803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.171537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.200399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.201986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.204988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.221504] Lustre: Skipped 1 previous similar message [ 996.322979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.339533] Lustre: Skipped 1 previous similar message [ 1001.445627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.453939] Lustre: Skipped 1 previous similar message [ 1002.976692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1003.143694] Lustre: server umount lustre-OST0000 complete [ 1010.666247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.681094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.692873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.407354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1021.929735] LustreError: 34975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.944021] LustreError: 34975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1023.552383] Lustre: server umount lustre-MDT0000 complete [ 1027.041585] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.067759] LustreError: 34978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1027.612664] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791830 with bad export cookie 8833617925443988018 [ 1027.614858] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.634368] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1028.004969] Lustre: server umount lustre-MDT0001 complete [ 1038.203784] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1040.625672] Key type lgssc unregistered [ 1040.943773] LNet: 42833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.952985] LNetError: 42833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.974183] LNet: Removed LNI 192.168.201.148@tcp [ 1041.713474] Key type .llcrypt unregistered [ 1041.715556] Key type ._llcrypt unregistered [ 1060.334308] Key type ._llcrypt registered [ 1060.337293] Key type .llcrypt registered [ 1060.424243] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 1074.334968] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1075.237822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.513498] alg: No test for adler32 (adler32-zlib) [ 1076.562981] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1076.844354] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1078.511294] Key type lgssc registered [ 1079.585956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.275834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.319408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.589892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1116.612340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1116.664442] Lustre: lustre-MDT0000: new disk, initializing [ 1116.733709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.756371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1122.207457] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1134.228482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.494185] Lustre: 45984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1134.589667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1134.592610] Lustre: Skipped 1 previous similar message [ 1134.654832] Lustre: lustre-MDT0001: new disk, initializing [ 1134.800618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.860904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1134.874798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1140.895522] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1151.853901] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.541661] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1166.614877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1166.973523] Lustre: lustre-OST0000: new disk, initializing [ 1166.983731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1166.999403] Lustre: 47955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1167.132432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.976602] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1176.132047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1176.143835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1176.252717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1181.096214] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1184.892233] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1185.095313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1188.947618] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1189.147717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1191.911840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.914757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.915681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.939373] Lustre: Skipped 1 previous similar message [ 1191.970197] LustreError: Skipped 1 previous similar message [ 1197.027626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.034414] Lustre: Skipped 2 previous similar messages [ 1202.157693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.165539] Lustre: Skipped 1 previous similar message [ 1205.215537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1205.348872] Lustre: server umount lustre-OST0000 complete [ 1211.872684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1211.882272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.889740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.367875] Lustre: server umount lustre-MDT0000 complete [ 1218.712147] LustreError: 47953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792021 with bad export cookie 4723674196368142352 [ 1218.716783] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.718405] LustreError: 47953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1219.074481] Lustre: server umount lustre-MDT0001 complete [ 1227.609549] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:53:50 (1787792030) [ 1234.154759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.572276] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.591866] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1234.657542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.180226] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1240.041361] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.141375] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.885800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.355689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.852351] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1260.426804] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.379964] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1272.908414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.251352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.752350] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1288.328363] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.941407] LustreError: 52944:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1293.819336] Lustre: Modifying parameter general.timeout=1 in log params [ 1294.856117] Lustre: Modifying parameter general.timeout=2 in log params [ 1297.433340] Lustre: Modifying parameter general.timeout=5 in log params [ 1297.437252] Lustre: Skipped 2 previous similar messages [ 1301.908323] Lustre: Modifying parameter general.timeout=7 in log params [ 1301.915708] Lustre: Skipped 1 previous similar message [ 1310.247741] Lustre: Modifying parameter general.timeout=16 in log params [ 1310.252826] Lustre: Skipped 8 previous similar messages [ 1326.666350] Lustre: Modifying parameter general.timeout=36 in log params [ 1326.672313] Lustre: Skipped 19 previous similar messages [ 1359.503882] Lustre: Modifying parameter general.timeout=77 in log params [ 1359.513975] Lustre: Skipped 40 previous similar messages [ 1418.461822] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:57:01 (1787792221) [ 1421.562862] Lustre: 59882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1423.400541] Lustre: 59980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1424.242271] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1424.252190] Lustre: Skipped 56 previous similar messages [ 1427.242416] Lustre: 60193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1427.266029] Lustre: 60193:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1462.080832] Lustre: 61757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1462.096678] Lustre: 61757:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1462.152433] LustreError: 61757:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1467.089486] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:57:49 (1787792269) [ 1483.746271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1483.760678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.787878] Lustre: Skipped 1 previous similar message [ 1483.804367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.814870] Lustre: Skipped 2 previous similar messages [ 1485.798061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.939596] Lustre: server umount lustre-MDT0000 complete [ 1490.924668] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.956322] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1492.394592] LustreError: 50395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792295 with bad export cookie 4723674196368143227 [ 1492.399509] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.410962] LustreError: 50395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1492.688481] Lustre: server umount lustre-MDT0001 complete [ 1506.847722] Lustre: server umount lustre-OST0000 complete [ 1532.926434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.185466] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1533.410987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.218841] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1545.969736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.069465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1546.104418] Lustre: 64290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1551.227584] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1559.636342] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1565.375538] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1572.764588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.969814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1573.291487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.302243] Lustre: Skipped 1 previous similar message [ 1579.198785] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1587.404936] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.875896] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1588.880618] Lustre: Skipped 19 previous similar messages [ 1588.947942] Lustre: 66855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1598.437653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.446299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.461618] Lustre: Skipped 1 previous similar message [ 1598.477083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.488251] Lustre: Skipped 3 previous similar messages [ 1603.374907] Lustre: server umount lustre-OST0000 complete [ 1608.165030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.168335] LustreError: Skipped 1 previous similar message [ 1608.174924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.185456] Lustre: Skipped 1 previous similar message [ 1614.308588] LustreError: 65201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.379846] Lustre: server umount lustre-MDT0000 complete [ 1618.308484] LustreError: 65041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792421 with bad export cookie 4723674196368146244 [ 1618.309259] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.332794] LustreError: 65041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1618.812028] Lustre: server umount lustre-MDT0001 complete [ 1629.846242] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1632.522941] Key type lgssc unregistered [ 1632.810641] LNet: 68073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.818709] LNetError: 68073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.845793] LNet: Removed LNI 192.168.201.148@tcp [ 1633.730870] Key type .llcrypt unregistered [ 1633.737297] Key type ._llcrypt unregistered [ 1656.426626] Key type ._llcrypt registered [ 1656.429208] Key type .llcrypt registered [ 1656.520826] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1657.878198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1658.052448] alg: No test for adler32 (adler32-zlib) [ 1659.157466] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1659.337290] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1661.031359] Key type lgssc registered [ 1662.199562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1669.362336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1669.375302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.717206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.649389] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1680.863993] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.879039] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1683.486228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.950813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.987918] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1695.168943] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1700.511643] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1706.974201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.735755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.391452] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1724.363954] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1733.184290] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1745.377036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.386449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.398066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.918061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.921418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.936354] Lustre: Skipped 1 previous similar message [ 1746.948673] Lustre: Skipped 2 previous similar messages [ 1749.536604] Lustre: server umount lustre-MDT0000 complete [ 1752.038469] LustreError: 69498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.938991] LustreError: 72182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792556 with bad export cookie 5862114952300955453 [ 1753.944357] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.953995] LustreError: 72182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.362946] Lustre: server umount lustre-MDT0001 complete [ 1769.498026] Lustre: server umount lustre-OST0000 complete [ 1796.540111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.865267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1796.892489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1797.298477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.136911] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1811.565175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.833157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.889177] Lustre: 74370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.498685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.760930] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1826.110187] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.365112] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.914622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.080710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1840.598470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.874592] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1857.723241] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.862648] Lustre: 76947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1861.739546] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1870.817408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.829444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.848967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.505416] Lustre: server umount lustre-OST0000 complete [ 1879.009717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1879.027347] LustreError: Skipped 1 previous similar message [ 1879.040095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.059512] Lustre: Skipped 1 previous similar message [ 1879.063234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.073570] Lustre: Skipped 2 previous similar messages [ 1883.519882] Lustre: server umount lustre-MDT0000 complete [ 1886.694438] LustreError: 74381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.701822] LustreError: 74381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1888.540842] LustreError: 76238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792691 with bad export cookie 5862114952300956300 [ 1888.542530] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.551850] LustreError: 76238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.229786] Lustre: server umount lustre-MDT0001 complete [ 1904.567418] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1908.533862] Key type lgssc unregistered [ 1909.023769] LNet: 78164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.038990] LNetError: 78164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.123811] LNet: Removed LNI 192.168.201.148@tcp [ 1911.253168] Key type .llcrypt unregistered [ 1911.255946] Key type ._llcrypt unregistered [ 1936.368522] Key type ._llcrypt registered [ 1936.371459] Key type .llcrypt registered [ 1936.488876] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1937.783749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.879900] alg: No test for adler32 (adler32-zlib) [ 1939.085382] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1939.388400] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1941.095396] Key type lgssc registered [ 1942.514776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.956401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.976504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.475338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.868258] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1961.955443] LustreError: 79579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.965736] LustreError: 79579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1963.733188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.443736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1970.120943] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1979.615777] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1986.578624] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1993.745146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1994.630647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.594106] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2011.021121] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.235477] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2018.496211] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.457050] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.468233] Lustre: Skipped 1 previous similar message [ 2023.642930] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2023.646366] Lustre: Skipped 2 previous similar messages [ 2027.733798] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.740498] Lustre: Skipped 2 previous similar messages [ 2036.272483] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2036.275750] Lustre: Skipped 7 previous similar messages [ 2053.022821] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2053.034424] Lustre: Skipped 15 previous similar messages [ 2085.231222] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2085.237742] Lustre: Skipped 28 previous similar messages [ 2137.340502] Lustre: 87693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2140.669452] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:09:02 (1787792942) [ 2142.433245] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2144.481324] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:09:06 (1787792946) [ 2151.397078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.403688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.421282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.445766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.449363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.468221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.562230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.564638] Lustre: Skipped 2 previous similar messages [ 2163.168616] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2163.505258] Lustre: server umount lustre-OST0000 complete [ 2168.804080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.809801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.827720] Lustre: Skipped 1 previous similar message [ 2173.922207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.930226] Lustre: Skipped 2 previous similar messages [ 2183.138165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.139775] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2183.146053] Lustre: Skipped 3 previous similar messages [ 2183.442578] Lustre: server umount lustre-MDT0000 complete [ 2188.270945] LustreError: 79579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.295865] LustreError: 79579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2189.427432] LustreError: 83904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792992 with bad export cookie 13868050778277591961 [ 2189.430868] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.435851] LustreError: 83904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2190.039956] Lustre: server umount lustre-MDT0001 complete [ 2202.402751] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2206.519365] Key type lgssc unregistered [ 2206.748785] LNet: 88948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.760738] LNetError: 88948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.787784] LNet: Removed LNI 192.168.201.148@tcp [ 2208.107049] Key type .llcrypt unregistered [ 2208.109388] Key type ._llcrypt unregistered [ 2215.137815] Key type ._llcrypt registered [ 2215.142861] Key type .llcrypt registered [ 2215.367152] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2221.654845] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules [ 2223.973801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.005396] alg: No test for adler32 (adler32-zlib) [ 2225.462635] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2225.974181] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2227.775206] Key type lgssc registered [ 2228.861918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2230.699177] LustreError: 90188:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2252.425316] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2255.114903] LustreError: 90188:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2266.673541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2266.715578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.643797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.953709] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2278.887654] LustreError: 91234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.915671] LustreError: 91234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2282.258367] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:11:24 (1787793084) [ 2284.000653] LustreError: 91233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.914368] LustreError: 91265:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2286.929286] LustreError: 91265:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2286.937725] LustreError: 91265:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2287.288311] Lustre: server umount lustre-MDT0000 complete [ 2301.453273] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2304.457728] Key type lgssc unregistered [ 2304.735129] LNet: 92762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.754396] LNetError: 92762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.773285] LNet: Removed LNI 192.168.201.148@tcp [ 2305.646047] Key type .llcrypt unregistered [ 2305.651328] Key type ._llcrypt unregistered [ 2324.874411] Key type ._llcrypt registered [ 2324.878711] Key type .llcrypt registered [ 2324.982105] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2326.698084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.786129] alg: No test for adler32 (adler32-zlib) [ 2327.853911] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2328.063989] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2329.730634] Key type lgssc registered [ 2331.667265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2341.872499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.906534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.817241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.074430] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2353.140323] LustreError: 93981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.167677] LustreError: 93981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2358.268850] LustreError: 93982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.511280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.470773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2366.517368] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2375.884401] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2383.036576] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.915694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.922868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.090567] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2406.526171] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2464.803213] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:14:27 (1787793267) [ 2466.978815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2468.983976] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:14:31 (1787793271) [ 2474.980251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.989671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.002476] Lustre: Skipped 2 previous similar messages [ 2475.022021] Lustre: Skipped 2 previous similar messages [ 2479.458392] Lustre: server umount lustre-MDT0000 complete [ 2480.099789] LustreError: 93982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.119027] LustreError: 93982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2483.197446] LustreError: 93962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793286 with bad export cookie 3892039798943966509 [ 2483.207024] LustreError: 93962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2483.208884] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.439586] Lustre: server umount lustre-MDT0001 complete [ 2497.574776] Lustre: server umount lustre-OST0000 complete [ 2511.128653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.784641] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.968496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.969305] LustreError: 98301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.142711] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2526.189943] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.223194] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2527.681102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.759783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.838711] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2544.359272] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2550.997218] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.401431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.661291] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2567.668158] LustreError: 98290:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2567.695628] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2567.707971] LustreError: 100384:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2567.717414] LustreError: 100384:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2567.723205] LustreError: 100384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2567.826714] Lustre: server umount lustre-OST0000 complete [ 2567.829847] LustreError: 100384:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2577.485365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.792151] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2577.808600] LustreError: 98288:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2577.834632] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2577.843786] LustreError: 100851:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2577.856474] LustreError: 100851:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2577.873049] LustreError: 100851:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2577.971300] Lustre: server umount lustre-OST0000 complete [ 2577.974815] LustreError: 100851:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2590.881360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.155962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2591.170860] Lustre: Found index 0 for lustre-OST0000, updating log [ 2591.176150] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2591.334351] Lustre: lustre-OST0000: new disk, initializing [ 2591.342852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2591.350706] Lustre: 101388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2591.854628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.304326] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2605.088567] Lustre: Failing over lustre-OST0000 [ 2605.271657] Lustre: server umount lustre-OST0000 complete [ 2610.144630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2610.160643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.176830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.193607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.196449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.086714] Lustre: server umount lustre-MDT0000 complete [ 2617.317944] LustreError: 98301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.344674] LustreError: 98301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2620.849120] LustreError: 100404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793423 with bad export cookie 3892039798943969253 [ 2620.855648] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.340209] Lustre: server umount lustre-MDT0001 complete [ 2641.852131] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:17:24 (1787793444) [ 2665.757548] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2678.234170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.750278] LustreError: 103736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.916613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.786983] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2696.257416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.634217] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2705.943237] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2714.128315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.875707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.877856] Lustre: Skipped 1 previous similar message [ 2714.902254] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2721.003943] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2721.033031] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2721.035798] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2722.670153] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2734.202724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2734.360613] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2734.485453] Lustre: lustre-OST0001: new disk, initializing [ 2734.489776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2734.500962] Lustre: 106055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2741.977829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2742.015520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2742.168857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2743.544938] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2753.005469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2762.810490] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2769.386784] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.397536] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.428079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2769.438529] Lustre: Skipped 1 previous similar message [ 2774.682187] Lustre: server umount lustre-MDT0001 complete [ 2778.004241] LustreError: 105126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.026517] LustreError: 105126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2786.651716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.840420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2786.852057] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2786.985791] Lustre: 103724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2787.980463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.985827] Lustre: Skipped 1 previous similar message [ 2788.502739] LustreError: 107596:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2788.513055] LustreError: 107596:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2788.524477] LustreError: 103728:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2788.659278] LustreError: 107601:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -17 [ 2788.672436] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2788.672436] [ 2793.488375] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2794.056487] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2806.688485] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:20:08 (1787793608) [ 2827.315581] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2848.779534] Lustre: 109672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2874.985846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2879.162434] Lustre: 110807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2972.656590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.674636] Lustre: Skipped 4 previous similar messages [ 2972.675068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.690288] Lustre: Skipped 7 previous similar messages [ 2978.530081] Lustre: server umount lustre-MDT0000 complete [ 2982.889836] LustreError: 103732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.921907] LustreError: 103732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2983.337242] LustreError: 103717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793786 with bad export cookie 3892039798943970254 [ 2983.342644] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.347362] LustreError: 103717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2983.844234] Lustre: server umount lustre-MDT0001 complete [ 2999.129891] Lustre: server umount lustre-OST0000 complete [ 3013.953505] Lustre: server umount lustre-OST0001 complete [ 3019.801580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3029.749899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3038.566201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3050.284504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3073.634761] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3086.442175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.459024] Lustre: lustre-MDT0000: reset Object Index mappings [ 3087.252285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.070703] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3103.630961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.652538] Lustre: lustre-MDT0001: reset Object Index mappings [ 3110.513421] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3114.297332] Lustre: 115916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3121.798209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.646327] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3129.898777] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3132.928863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3139.594755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.735340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3145.753410] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3148.371994] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3156.993423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3161.148453] Lustre: 117792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3213.512710] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:26:55 (1787794015) [ 3242.465131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3242.476729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.488520] Lustre: Skipped 1 previous similar message [ 3242.495501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.507301] Lustre: Skipped 4 previous similar messages [ 3246.835299] Lustre: server umount lustre-MDT0000 complete [ 3248.099985] LustreError: 114768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.110207] LustreError: 114768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3251.222603] LustreError: 114748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794054 with bad export cookie 3892039798944017126 [ 3251.248154] LustreError: 114748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3251.261250] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.779250] Lustre: server umount lustre-MDT0001 complete [ 3268.780259] Lustre: server umount lustre-OST0000 complete [ 3279.078184] Lustre: server umount lustre-OST0001 complete [ 3286.489569] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3298.761274] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3340.735496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.932589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3340.962123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3341.031735] Lustre: lustre-MDT0000: new disk, initializing [ 3341.104859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3345.925729] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3354.553193] Lustre: server umount lustre-MDT0000 complete [ 3367.635533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.111601] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3368.149549] Lustre: 122067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3368.169292] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3368.484780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.497328] Lustre: Skipped 4 previous similar messages [ 3373.312491] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3379.039132] Lustre: server umount lustre-MDT0000 complete [ 3389.042535] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:29:50 (1787794190) [ 3390.854794] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3392.939243] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:29:55 (1787794195) [ 3412.944614] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3424.681986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.025539] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3439.909326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.093158] Lustre: 124117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3440.191981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3440.203372] Lustre: Skipped 1 previous similar message [ 3440.266474] Lustre: lustre-MDT0001: new disk, initializing [ 3440.452912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3440.467424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3444.851929] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3450.328580] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3456.927738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.265204] Lustre: lustre-OST0000: new disk, initializing [ 3457.267318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3457.279867] Lustre: 125694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3458.544759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3458.561727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.756890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3463.745583] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3474.356751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3474.689213] Lustre: lustre-OST0001: new disk, initializing [ 3474.708440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3474.723294] Lustre: 126564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3476.062623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3476.071789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3476.235748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3483.170591] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3495.995152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3500.200540] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3900.738421] Lustre: 132981:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3900.751772] Lustre: 132981:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3900.757463] Lustre: 132981:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3900.762344] Lustre: 132981:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3900.778430] Lustre: 132981:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3900.784174] Lustre: 132981:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4171.328333] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:42:53 (1787794973) [ 4188.131938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.138544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4188.148109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.148120] Lustre: Skipped 7 previous similar messages [ 4188.153992] Lustre: Skipped 6 previous similar messages [ 4192.625524] Lustre: server umount lustre-MDT0000 complete [ 4193.256390] LustreError: 127369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.287671] LustreError: 127369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4196.182022] LustreError: 124855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794999 with bad export cookie 3892039798944085173 [ 4196.185951] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.188659] LustreError: 124855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4198.371326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.382221] Lustre: Skipped 3 previous similar messages [ 4198.388723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4198.390547] Lustre: Skipped 3 previous similar messages [ 4202.948462] Lustre: server umount lustre-MDT0001 complete [ 4213.836241] Lustre: server umount lustre-OST0000 complete [ 4223.264571] Lustre: server umount lustre-OST0001 complete [ 4230.473806] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4242.694209] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4284.654099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.955662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4284.977331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4285.097679] Lustre: lustre-MDT0000: new disk, initializing [ 4285.250880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.267690] Lustre: Skipped 4 previous similar messages [ 4285.291542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4290.799886] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4301.990484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.239722] Lustre: 136032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4302.342046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4302.349714] Lustre: Skipped 1 previous similar message [ 4302.467752] Lustre: lustre-MDT0001: new disk, initializing [ 4302.684822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4302.701141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4309.086441] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4319.387670] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.031946] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4332.452828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.807801] Lustre: lustre-OST0000: new disk, initializing [ 4332.811543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4332.830292] Lustre: 138002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4333.967220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4333.989472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4334.170352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4340.285771] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4350.061531] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4359.444637] Lustre: lustre-MDD0000: changelog on [ 4362.996259] Lustre: lustre-MDD0001: changelog on [ 6024.067931] Lustre: lustre-MDD0001: changelog off [ 6026.417431] Lustre: 138680:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6027.237580] Lustre: lustre-MDD0000: changelog off [ 6033.889610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.890665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.903923] LustreError: Skipped 1 previous similar message [ 6033.920438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.929111] Lustre: Skipped 3 previous similar messages [ 6039.013944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6039.022454] Lustre: Skipped 1 previous similar message [ 6039.513633] Lustre: server umount lustre-OST0000 complete [ 6043.617409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.625878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.635847] Lustre: Skipped 1 previous similar message [ 6043.643359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6049.249454] LustreError: 136043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.262892] LustreError: 136043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6049.276118] Lustre: server umount lustre-MDT0000 complete [ 6052.618153] LustreError: 136022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796855 with bad export cookie 3892039798944906861 [ 6052.623833] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6052.627089] LustreError: 136022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6053.070717] Lustre: server umount lustre-MDT0001 complete [ 6064.930620] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6067.437402] Key type lgssc unregistered [ 6067.744789] LNet: 147125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6067.757720] LNetError: 147125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6068.772978] LNet: Removed LNI 192.168.201.148@tcp [ 6069.519321] Key type .llcrypt unregistered [ 6069.522540] Key type ._llcrypt unregistered [ 6087.458989] Key type ._llcrypt registered [ 6087.460813] Key type .llcrypt registered [ 6087.539814] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6100.616253] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6101.796852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6101.819509] alg: No test for adler32 (adler32-zlib) [ 6102.813473] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 6103.026912] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 6104.663177] Key type lgssc registered [ 6105.799132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6140.055558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6140.078479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.234703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6141.253944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6141.320137] Lustre: lustre-MDT0000: new disk, initializing [ 6141.369395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.384616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6145.741491] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6156.834246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.031806] Lustre: 150319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6157.107927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6157.121154] Lustre: Skipped 1 previous similar message [ 6157.251723] Lustre: lustre-MDT0001: new disk, initializing [ 6157.351662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6157.383736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6157.392117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6161.551111] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6172.006984] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6177.311884] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6183.633348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.874828] Lustre: lustre-OST0000: new disk, initializing [ 6183.879688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6183.886173] Lustre: 152288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6183.967645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.343485] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6191.692959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6191.701817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6191.826123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6200.091198] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.119652] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module obdecho/obdecho [ 6208.037603] LustreError: 153291:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6208.043865] LustreError: 153291:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6209.617331] LustreError: 153387:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6222.819783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.833267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.069509] Lustre: server umount lustre-MDT0000 complete [ 6229.575415] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797032 with bad export cookie 2258936096336439035 [ 6229.580641] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.587579] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6230.045450] Lustre: server umount lustre-MDT0001 complete [ 6244.108675] Lustre: server umount lustre-OST0000 complete [ 6263.828863] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6273.270885] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6312.927151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.302406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6313.374572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6313.562501] Lustre: lustre-MDT0000: new disk, initializing [ 6313.692399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.731574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6319.452968] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6331.552784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.791823] Lustre: 157106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6331.892220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6331.899625] Lustre: Skipped 1 previous similar message [ 6332.024133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6332.035852] Lustre: Skipped 2 previous similar messages [ 6332.070155] Lustre: lustre-MDT0001: new disk, initializing [ 6332.336395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6332.467684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6332.498571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6336.041186] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6345.350450] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.589748] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.832813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.069750] Lustre: lustre-OST0000: new disk, initializing [ 6359.073814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6359.077785] Lustre: 159073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6359.150449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.850551] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6368.414420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6368.432390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6368.633163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6378.005772] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6382.888499] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6383.104728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6388.606327] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6388.925745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6394.339821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.340378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.343015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.347692] Lustre: Skipped 3 previous similar messages [ 6397.222252] Lustre: server umount lustre-OST0000 complete [ 6404.065217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6404.076802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.089799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.096909] Lustre: Skipped 1 previous similar message [ 6407.540797] Lustre: server umount lustre-MDT0000 complete [ 6409.706076] LustreError: 157117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.733130] LustreError: 157117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6411.648417] LustreError: 157099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797214 with bad export cookie 2258936096336439994 [ 6411.652030] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.655573] LustreError: 157099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6412.083786] Lustre: server umount lustre-MDT0001 complete [ 6415.650621] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:20:18 (1787797218) [ 6417.263122] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6418.949361] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:20:21 (1787797221) [ 6426.736643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.207398] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6427.275405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.021585] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6432.736689] LustreError: 161487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.860820] LustreError: 161486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.770146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.277218] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6456.534216] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.733920] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.598330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6469.984424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.991507] Lustre: Skipped 1 previous similar message [ 6476.603242] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6484.304414] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.878094] LustreError: 161513:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -5 [ 6496.742799] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.763305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.782661] Lustre: Skipped 3 previous similar messages [ 6496.787707] Lustre: Skipped 2 previous similar messages [ 6502.293567] Lustre: server umount lustre-MDT0000 complete [ 6505.832478] LustreError: 161482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6506.394137] LustreError: 161467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797309 with bad export cookie 2258936096336440876 [ 6506.400350] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.403636] LustreError: 161467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6506.976738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.991643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6506.996457] Lustre: Skipped 5 previous similar messages [ 6512.894847] Lustre: server umount lustre-MDT0001 complete [ 6520.975321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6534.877638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.100791] LustreError: 165332:0:(llog.c:1655:llog_backup()) MGC192.168.201.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6535.113694] Lustre: 165332:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6536.675443] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.697676] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6552.933935] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.950465] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6587.745730] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.773563] LustreError: 165337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6596.576667] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6596.581600] Lustre: 165332:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6637.571381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6642.382577] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6642.664787] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6652.256814] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.283589] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 6652.604233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.911512] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1f595a6948710762 [ 6652.920976] Lustre: MGC192.168.201.148@tcp: Connection restored to 0@lo (at 0@lo) [ 6658.199879] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6658.550690] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6666.979409] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6669.201424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6675.722686] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6677.288708] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6693.323037] Lustre: server umount lustre-MDT0000 complete [ 6694.371873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.378947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6697.630424] LustreError: 161468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797500 with bad export cookie 2258936096336447330 [ 6697.637172] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.642197] LustreError: 161468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6697.933994] Lustre: server umount lustre-MDT0001 complete [ 6712.099797] Lustre: server umount lustre-OST0000 complete [ 6731.659756] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6741.064969] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6777.893839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.175863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6778.209361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6778.324455] Lustre: lustre-MDT0000: new disk, initializing [ 6778.452277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.467383] Lustre: Skipped 1 previous similar message [ 6778.486468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6783.808795] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6795.745359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.868813] Lustre: 170436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6796.163805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6801.466078] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6811.844169] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6817.391414] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6823.874513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6824.096514] Lustre: lustre-OST0000: new disk, initializing [ 6824.099444] Lustre: Skipped 1 previous similar message [ 6824.102103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6824.105577] Lustre: Skipped 2 previous similar messages [ 6824.109974] Lustre: 172402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6830.575951] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6831.171425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6831.190802] Lustre: Skipped 1 previous similar message [ 6831.200963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6831.319745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6840.261780] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6844.580905] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6844.822701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6849.471923] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6849.701493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6852.070671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6852.077484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.080188] LustreError: Skipped 1 previous similar message [ 6852.091553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.110562] Lustre: Skipped 3 previous similar messages [ 6852.117050] Lustre: Skipped 2 previous similar messages [ 6857.599596] Lustre: server umount lustre-OST0000 complete [ 6867.426696] LustreError: 170446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.442430] LustreError: 170446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6867.486889] Lustre: server umount lustre-MDT0000 complete [ 6871.220339] LustreError: 172401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797674 with bad export cookie 2258936096336464165 [ 6871.227160] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.234262] LustreError: 172401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6871.662117] Lustre: server umount lustre-MDT0001 complete [ 6881.396269] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:28:03 (1787797683) [ 6887.775954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.069525] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6901.247231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.561596] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6915.451732] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6921.842053] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.764949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.881953] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6944.995389] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6958.503762] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:29:20 (1787797760) [ 6963.169215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.187774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6963.194671] Lustre: Skipped 2 previous similar messages [ 6963.200433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6963.231630] Lustre: Skipped 6 previous similar messages [ 6968.502920] Lustre: server umount lustre-OST0000 complete [ 6973.427736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6979.651303] Lustre: server umount lustre-MDT0000 complete [ 6983.403578] LustreError: 174796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797786 with bad export cookie 2258936096336465040 [ 6983.414086] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.415678] LustreError: 174796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6984.091170] Lustre: server umount lustre-MDT0001 complete [ 6996.675351] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6999.610524] Key type lgssc unregistered [ 6999.850867] LNet: 178547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6999.872353] LNetError: 178547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6999.893506] LNet: Removed LNI 192.168.201.148@tcp [ 7001.014134] Key type .llcrypt unregistered [ 7001.017104] Key type ._llcrypt unregistered [ 7019.112631] Key type ._llcrypt registered [ 7019.115621] Key type .llcrypt registered [ 7019.262116] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7020.463047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7020.488049] alg: No test for adler32 (adler32-zlib) [ 7021.634238] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 7021.914191] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 7023.551228] Key type lgssc registered [ 7024.719500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7032.540739] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7032.557601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7048.159479] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7053.279335] LustreError: 179794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 7058.403524] LustreError: 179794:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7058.422919] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7058.429815] LustreError: MGC192.168.201.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7058.449346] LustreError: 179794:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7058.458816] LustreError: 179794:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7058.463840] LustreError: 179794:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7058.539565] Lustre: server umount lustre-OST0000 complete [ 7058.541875] LustreError: 179794:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7067.185144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.696898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7073.664255] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7078.890359] LustreError: 180295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.924272] LustreError: 180295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7082.140676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.516867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7087.235313] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7094.136908] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7099.565529] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7106.312500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7106.578495] LustreError: 182109:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7106.590870] Lustre: 182109:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7106.803377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.987801] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7120.182868] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7123.426120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7123.431328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.448517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.450619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.450722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.476626] Lustre: Skipped 1 previous similar message [ 7128.864184] Lustre: server umount lustre-MDT0000 complete [ 7129.576600] LustreError: 181023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.602326] LustreError: 181023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7132.296168] LustreError: 180277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797935 with bad export cookie 676962562673646244 [ 7132.312163] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.667851] Lustre: server umount lustre-MDT0001 complete [ 7146.564681] Lustre: server umount lustre-OST0000 complete [ 7155.829674] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7165.384372] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7199.404098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.745706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7199.772817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7199.879071] Lustre: lustre-MDT0000: new disk, initializing [ 7200.031464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.048746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7205.368979] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7217.547818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.675662] Lustre: 185633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7217.766741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7217.774419] Lustre: Skipped 1 previous similar message [ 7217.879596] Lustre: lustre-MDT0001: new disk, initializing [ 7218.075695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7218.151532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7218.166479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7223.594177] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7233.854861] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7240.109505] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7246.546256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7246.735755] Lustre: lustre-OST0000: new disk, initializing [ 7246.738572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7246.743953] Lustre: 187605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7246.809670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.677820] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7257.163342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7257.176238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7257.296059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7260.649953] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7265.944582] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7266.207878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7270.305634] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7270.534635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7272.930432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.932539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.940637] Lustre: Skipped 1 previous similar message [ 7272.948415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.956815] Lustre: Skipped 2 previous similar messages [ 7278.055349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.067259] Lustre: Skipped 1 previous similar message [ 7278.607480] Lustre: server umount lustre-OST0000 complete [ 7283.168980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.183133] Lustre: Skipped 2 previous similar messages [ 7283.190181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.289484] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.297803] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7288.354319] Lustre: server umount lustre-MDT0000 complete [ 7291.643924] LustreError: 185627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798094 with bad export cookie 676962562673646965 [ 7291.660935] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.662373] LustreError: 185627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7291.930609] Lustre: server umount lustre-MDT0001 complete [ 7300.138385] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:35:03 (1787798103) [ 7319.021514] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7322.352297] Key type lgssc unregistered [ 7322.699313] LNet: 190552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7322.722123] LNetError: 190552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7322.735316] LNet: Removed LNI 192.168.201.148@tcp [ 7323.412513] Key type .llcrypt unregistered [ 7323.418047] Key type ._llcrypt unregistered [ 7341.114409] Key type ._llcrypt registered [ 7341.117421] Key type .llcrypt registered [ 7341.239027] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7342.219611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7342.301062] alg: No test for adler32 (adler32-zlib) [ 7343.341391] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 7343.626635] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 7345.295182] Key type lgssc registered [ 7346.317863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7353.841066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7353.859705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.252365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.028841] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7365.605268] LustreError: 191770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.630259] LustreError: 191770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7366.412538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.649063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7370.735163] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7378.012783] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7383.153866] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7390.689633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.847520] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7390.851622] LustreError: 193633:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7390.857844] Lustre: 193633:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7391.008904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.067967] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7405.557529] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7407.595970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7407.611081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.631875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.680186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.690502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.699520] Lustre: server umount lustre-MDT0000 complete [ 7416.803143] LustreError: 191770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.814292] LustreError: 191770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7417.408769] LustreError: 191750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798220 with bad export cookie 9487687134730861160 [ 7417.425772] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.924867] Lustre: server umount lustre-MDT0001 complete [ 7433.077257] Lustre: server umount lustre-OST0000 complete [ 7442.389340] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7452.406813] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7484.439467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.640937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7484.661393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7484.713275] Lustre: lustre-MDT0000: new disk, initializing [ 7484.768766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.780146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7489.316374] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7499.186596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.443286] Lustre: 197159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7499.506878] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7499.509256] Lustre: Skipped 1 previous similar message [ 7499.514828] LustreError: 197940:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7499.524216] LustreError: 197940:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7499.528276] Lustre: 197940:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7499.547537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7499.551157] Lustre: Skipped 1 previous similar message [ 7499.611711] Lustre: lustre-MDT0001: new disk, initializing [ 7499.686794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.715620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7499.726989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7503.944928] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7512.923930] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7518.472334] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7525.796733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7526.004820] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7526.007477] Lustre: Skipped 1 previous similar message [ 7526.011395] LustreError: 199102:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7526.017467] LustreError: 199102:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7526.022473] Lustre: 199102:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7526.031161] Lustre: 199102:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7526.049186] Lustre: lustre-OST0000: new disk, initializing [ 7526.054203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7526.059883] Lustre: 199127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7526.161339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7531.809155] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7532.161801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7532.169117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7532.291916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7540.057732] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.297585] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7544.584230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7549.015205] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7549.233356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7553.009384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7553.013163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.018277] LustreError: Skipped 1 previous similar message [ 7553.025273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7553.032959] Lustre: Skipped 2 previous similar messages [ 7553.050031] Lustre: Skipped 3 previous similar messages [ 7558.120307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7558.128255] Lustre: Skipped 1 previous similar message [ 7563.241183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7563.255094] Lustre: Skipped 1 previous similar message [ 7565.281084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7565.472550] Lustre: server umount lustre-OST0000 complete [ 7571.424584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7571.434171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.458503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.585682] Lustre: server umount lustre-MDT0000 complete [ 7578.593882] LustreError: 197170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.606658] LustreError: 197170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7579.037872] LustreError: 197150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798382 with bad export cookie 9487687134730861888 [ 7579.041516] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.046501] LustreError: 197150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7579.321869] Lustre: server umount lustre-MDT0001 complete [ 7587.344535] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:39:49 (1787798389) [ 7600.936301] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7610.732420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.087934] LustreError: 202255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.139963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.583269] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7616.483165] LustreError: 202256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.183679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.487409] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7632.550329] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7639.617666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.710476] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7655.483961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.629214] Lustre: lustre-OST0001: new disk, initializing [ 7655.632979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7655.638684] Lustre: 204561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7655.728758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7655.735606] Lustre: Skipped 2 previous similar messages [ 7661.993499] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7663.176817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7663.186917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7663.363940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7672.251971] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7680.302551] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7682.379154] Lustre: DEBUG MARKER: ADD OST3 [ 7687.081846] loop: module loaded [ 7687.177340] loop0: detected capacity change from 0 to 204800000 [ 7687.575675] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7687.585483] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7687.625131] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7693.596668] loop0: detected capacity change from 0 to 204800000 [ 7694.509564] Lustre: DEBUG MARKER: STOP OST3 [ 7699.400660] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.578418] Lustre: lustre-OST0003: new disk, initializing [ 7699.583438] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7699.595107] Lustre: 206572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7701.387475] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7701.563056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7701.585055] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7701.637828] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7703.465702] LustreError: 206741:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7703.485323] Lustre: Failing over lustre-OST0003 [ 7703.724139] Lustre: server umount lustre-OST0003 complete [ 7706.570675] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7706.620651] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.633614] Lustre: Skipped 1 previous similar message [ 7706.646231] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.650487] LustreError: 203741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.676194] LustreError: 203741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7713.834995] Lustre: DEBUG MARKER: START OST3 again [ 7719.578492] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7719.775598] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7719.781883] Lustre: Skipped 1 previous similar message [ 7719.789468] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7721.396569] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7721.609281] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7721.610848] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7725.385851] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7745.783932] Lustre: Failing over lustre-OST0003 [ 7745.885212] Lustre: server umount lustre-OST0003 complete [ 7747.556809] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7747.557454] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.571955] LustreError: Skipped 1 previous similar message [ 7747.581038] LustreError: 204687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.588446] Lustre: Skipped 2 previous similar messages [ 7747.615580] LustreError: 204687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7765.992183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.996715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.007393] Lustre: Skipped 3 previous similar messages [ 7766.023914] Lustre: Skipped 5 previous similar messages [ 7779.295299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7779.603536] Lustre: server umount lustre-MDT0000 complete [ 7781.357741] LustreError: 202251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.380469] LustreError: 202251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7784.318149] LustreError: 202237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798587 with bad export cookie 9487687134730862770 [ 7784.328499] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7784.329048] LustreError: 202237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7790.970480] Lustre: server umount lustre-MDT0001 complete [ 7794.880369] Lustre: server umount lustre-OST0000 complete [ 7799.008590] Lustre: server umount lustre-OST0001 complete [ 7806.037851] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7814.544472] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7854.471308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.706490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7854.762421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7854.899954] Lustre: lustre-MDT0000: new disk, initializing [ 7854.970310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.000237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7859.225840] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7868.462357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.596912] Lustre: 212018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7868.913565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7872.689553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7882.324708] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7887.059469] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7892.904364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.175824] Lustre: lustre-OST0000: new disk, initializing [ 7893.181418] Lustre: Skipped 1 previous similar message [ 7893.190629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7893.201129] Lustre: Skipped 2 previous similar messages [ 7893.215543] Lustre: 213987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7894.514309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7894.528412] Lustre: Skipped 1 previous similar message [ 7894.539347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7894.686081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7898.734471] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7907.580132] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7912.810325] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7912.986425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7917.247894] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7917.490769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7920.102539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.105548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7920.125876] Lustre: Skipped 3 previous similar messages [ 7920.140172] Lustre: Skipped 13 previous similar messages [ 7925.470426] Lustre: server umount lustre-OST0000 complete [ 7930.348161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7935.134929] Lustre: server umount lustre-MDT0000 complete [ 7935.458047] LustreError: 213995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.473802] LustreError: 213995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7938.755959] LustreError: 213986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798741 with bad export cookie 9487687134730865066 [ 7938.761629] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7939.120673] Lustre: server umount lustre-MDT0001 complete [ 7942.412461] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:45:45 (1787798745) [ 7956.668131] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7964.876155] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8000.145996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.308699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8000.341801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8000.437789] Lustre: lustre-MDT0000: new disk, initializing [ 8000.549749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8004.764932] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8014.907631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.022284] Lustre: 218599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8015.206742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8019.203503] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8027.953563] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.199392] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.688458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.072132] Lustre: 220565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8041.801440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8046.365333] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8055.745716] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.597993] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8060.804616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8065.136509] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8065.370635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8067.553458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.567114] Lustre: Skipped 2 previous similar messages [ 8067.571930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8067.574740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8067.585121] Lustre: Skipped 5 previous similar messages [ 8081.375314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8081.518387] Lustre: server umount lustre-OST0000 complete [ 8091.418952] Lustre: server umount lustre-MDT0000 complete [ 8093.160716] LustreError: 218604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.179055] LustreError: 218604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8095.424994] LustreError: 218589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798898 with bad export cookie 9487687134730865948 [ 8095.432214] LustreError: 218589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8095.441807] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.830229] Lustre: server umount lustre-MDT0001 complete [ 8103.865132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.788735] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8119.798693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.253130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.258136] Lustre: Skipped 6 previous similar messages [ 8126.449751] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8135.515907] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8144.286630] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.119755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.819759] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8164.945224] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8185.319165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.326404] LustreError: Skipped 1 previous similar message [ 8191.017239] Lustre: server umount lustre-OST0000 complete [ 8195.556904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.570596] Lustre: Skipped 6 previous similar messages [ 8195.576468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.582719] Lustre: Skipped 12 previous similar messages [ 8200.474199] Lustre: server umount lustre-MDT0000 complete [ 8204.198315] LustreError: 222766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799007 with bad export cookie 9487687134730866830 [ 8204.208274] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.213492] LustreError: 222766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8204.534619] Lustre: server umount lustre-MDT0001 complete [ 8306.136692] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 8308.605421] Key type lgssc unregistered [ 8308.863436] LNet: 226471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8308.885629] LNetError: 226471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8308.901819] LNet: Removed LNI 192.168.201.148@tcp [ 8309.713372] Key type .llcrypt unregistered [ 8309.715229] Key type ._llcrypt unregistered [ 8334.193768] Key type ._llcrypt registered [ 8334.196879] Key type .llcrypt registered [ 8334.318178] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8349.471708] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8350.959091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8351.279317] alg: No test for adler32 (adler32-zlib) [ 8352.329254] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8352.546855] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 8354.191268] Key type lgssc registered [ 8355.395091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8392.482689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8392.518904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.902132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8393.954494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8394.109567] Lustre: lustre-MDT0000: new disk, initializing [ 8394.251193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.282684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8399.977655] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8411.780443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8411.895402] Lustre: 229833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8411.984179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8411.990681] Lustre: Skipped 1 previous similar message [ 8412.139278] Lustre: lustre-MDT0001: new disk, initializing [ 8412.284252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8412.329472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8412.342220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8417.136346] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8426.789877] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8431.752305] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8438.512158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8438.902504] Lustre: lustre-OST0000: new disk, initializing [ 8438.922673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8438.931599] Lustre: 231801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8439.068734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.313393] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8446.062530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8446.074838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8446.216313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8455.520888] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8460.139264] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8460.332788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8465.494392] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8465.734817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8472.038703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8472.042232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.052991] LustreError: Skipped 1 previous similar message [ 8472.061180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8472.069905] Lustre: Skipped 1 previous similar message [ 8474.441701] Lustre: server umount lustre-OST0000 complete [ 8478.688843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8478.701899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.716798] Lustre: Skipped 1 previous similar message [ 8478.724879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.275480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.276387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.304151] Lustre: Skipped 1 previous similar message [ 8484.475456] Lustre: server umount lustre-MDT0000 complete [ 8488.322710] LustreError: 229825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799291 with bad export cookie 3453959279719404882 [ 8488.332403] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.339102] LustreError: 229825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8488.900434] Lustre: server umount lustre-MDT0001 complete [ 8509.382755] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8521.539283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.109097] LustreError: 234735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.144196] LustreError: 234735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8522.197106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.335907] LustreError: 234736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.376992] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8532.454656] LustreError: 234735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.629294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.122747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8541.387788] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8544.836678] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8552.317592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8552.731466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8559.711217] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8568.741275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.958632] Lustre: lustre-OST0001: new disk, initializing [ 8568.962684] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8568.969131] Lustre: 237045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8569.125662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8576.611860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8576.637604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8576.750185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8576.996762] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8588.911494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8598.186051] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8611.999585] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:56:54 (1787799414) [ 8617.955962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.960147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8617.968652] Lustre: Skipped 3 previous similar messages [ 8617.973869] Lustre: Skipped 2 previous similar messages [ 8620.497617] Lustre: server umount lustre-MDT0000 complete [ 8623.079670] LustreError: 234731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8623.110225] LustreError: 234731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8624.211827] LustreError: 234717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799427 with bad export cookie 3453959279719405764 [ 8624.218239] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8624.237867] LustreError: 234717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8624.744579] Lustre: server umount lustre-MDT0001 complete [ 8638.111578] Lustre: server umount lustre-OST0000 complete [ 8651.307771] Lustre: server umount lustre-OST0001 complete [ 8658.298872] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8668.674386] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8703.819390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.179459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8704.220911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8704.351877] Lustre: lustre-MDT0000: new disk, initializing [ 8704.465608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.501505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8709.256620] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8721.234629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.413510] Lustre: 240993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8721.488606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8721.492529] Lustre: Skipped 1 previous similar message [ 8721.565347] Lustre: lustre-MDT0001: new disk, initializing [ 8721.699369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8721.718924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8727.265733] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8737.775581] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8744.466161] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8751.835594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8752.036842] Lustre: lustre-OST0000: new disk, initializing [ 8752.039974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8752.044971] Lustre: 242960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8753.660955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8753.673038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8753.826861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8758.226360] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8768.312241] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8773.667558] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8773.949521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8778.158974] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8778.448600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8784.354162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8784.357075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.362659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.378901] Lustre: Skipped 1 previous similar message [ 8784.391963] Lustre: Skipped 2 previous similar messages [ 8786.453676] Lustre: server umount lustre-OST0000 complete [ 8793.569299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8793.573178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.587347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.319197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8804.835526] LustreError: 240998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.865254] LustreError: 240998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8804.920819] Lustre: server umount lustre-MDT0000 complete [ 8808.808757] LustreError: 244299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799611 with bad export cookie 3453959279719407899 [ 8808.811121] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8808.815635] LustreError: 244299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8809.367756] Lustre: server umount lustre-MDT0001 complete [ 8818.504060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.285343] LustreError: 245178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8819.426355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.430270] Lustre: Skipped 2 previous similar messages [ 8824.705457] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8833.314349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.806565] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8844.219531] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8849.933996] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8856.318736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.067427] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8869.994209] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9017.312369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9017.319452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.341731] Lustre: Skipped 1 previous similar message [ 9017.348239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9017.356506] Lustre: Skipped 4 previous similar messages [ 9023.376939] Lustre: server umount lustre-OST0000 complete [ 9028.064539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9028.068241] LustreError: Skipped 1 previous similar message [ 9041.375135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.603528] Lustre: server umount lustre-MDT0000 complete [ 9043.939109] LustreError: 246242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9043.946737] LustreError: 246242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9046.876276] LustreError: 245158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799849 with bad export cookie 3453959279719408781 [ 9046.888844] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.906072] LustreError: 245158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9047.311764] Lustre: server umount lustre-MDT0001 complete [ 9137.356594] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 9140.196996] Key type lgssc unregistered [ 9140.444322] LNet: 248799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9140.460351] LNetError: 248799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9141.488285] LNet: Removed LNI 192.168.201.148@tcp [ 9142.472705] Key type .llcrypt unregistered [ 9142.475562] Key type ._llcrypt unregistered [ 9168.793910] Key type ._llcrypt registered [ 9168.795231] Key type .llcrypt registered [ 9168.903041] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9184.604195] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9185.546386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9185.611754] alg: No test for adler32 (adler32-zlib) [ 9186.634129] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 9186.827751] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 9188.552909] Key type lgssc registered [ 9189.376782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9237.265787] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9248.449910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9248.484131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.789087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.822394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.895875] Lustre: lustre-MDT0000: new disk, initializing [ 9249.989627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9250.009727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9254.937439] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9265.268453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.407472] Lustre: 252871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9265.463262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9265.466676] Lustre: Skipped 1 previous similar message [ 9265.573564] Lustre: lustre-MDT0001: new disk, initializing [ 9265.649800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9265.685227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9265.696498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9271.005918] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9276.602896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9283.295883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9283.619531] Lustre: lustre-OST0000: new disk, initializing [ 9283.623739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9283.632812] Lustre: 254504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9283.730305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.748816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9288.760542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9288.893412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9290.007602] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9301.003293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.229794] Lustre: lustre-OST0001: new disk, initializing [ 9301.232943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9301.240965] Lustre: 255374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9301.339570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9307.644221] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9308.264804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9308.277626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9308.379874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9318.359142] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9324.662309] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9334.253418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.260074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9334.263700] Lustre: Skipped 1 previous similar message [ 9339.365982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.374787] Lustre: Skipped 6 previous similar messages [ 9339.936088] Lustre: server umount lustre-MDT0000 complete [ 9344.182297] LustreError: 252862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800147 with bad export cookie 14229415203380887038 [ 9344.187897] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9344.199986] LustreError: 252862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9344.455699] Lustre: server umount lustre-MDT0001 complete [ 9358.240262] Lustre: server umount lustre-OST0000 complete [ 9360.864130] Lustre: 250374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787800147/real 1787800147] req@ffff8aed75570780 x1874644161341824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787800163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9360.893393] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.898568] Lustre: Skipped 2 previous similar messages [ 9362.262983] Lustre: server umount lustre-OST0001 complete [ 9374.855530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9385.171923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9393.600268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9404.861224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9424.891287] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9434.493747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.531192] Lustre: lustre-MDT0000: reset Object Index mappings [ 9435.291590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9439.518855] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9440.233701] LustreError: 260384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.245969] LustreError: 260384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9445.350541] LustreError: 260383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.101324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9448.128506] Lustre: lustre-MDT0001: reset Object Index mappings [ 9448.513831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9453.396666] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9456.373188] Lustre: 261525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9463.519904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9469.665355] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9473.002547] LustreError: 261877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9477.479678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9477.758833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9477.763825] Lustre: Skipped 1 previous similar message [ 9480.833160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9480.835348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9484.837650] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9492.720376] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9496.616864] Lustre: 263395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9501.677011] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.694510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.701395] Lustre: Skipped 1 previous similar message [ 9501.716245] Lustre: Skipped 3 previous similar messages [ 9506.786373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9506.796392] Lustre: Skipped 3 previous similar messages [ 9507.504516] Lustre: server umount lustre-MDT0000 complete [ 9511.194302] LustreError: 260363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800314 with bad export cookie 14229415203380893828 [ 9511.197776] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9511.206841] LustreError: 260363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9511.468341] Lustre: server umount lustre-MDT0001 complete [ 9524.082719] Lustre: server umount lustre-OST0000 complete [ 9527.264720] Lustre: 250376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787800314/real 1787800314] req@ffff8aec4d9c2940 x1874644161389184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787800330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9527.282306] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.301055] Lustre: Skipped 2 previous similar messages [ 9527.636773] Lustre: server umount lustre-OST0001 complete [ 9548.300540] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:12:30 (1787800350) [ 9563.418331] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9572.745744] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9606.042842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.283416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9606.322709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9606.391695] Lustre: lustre-MDT0000: new disk, initializing [ 9606.459521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9606.471204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.654958] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9621.801621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.880818] Lustre: 267649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9621.933161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9621.936957] Lustre: Skipped 1 previous similar message [ 9621.995499] Lustre: lustre-MDT0001: new disk, initializing [ 9622.117465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9622.131295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9626.961599] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9636.664407] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9642.937866] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9649.945302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9650.364914] Lustre: lustre-OST0000: new disk, initializing [ 9650.371153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9650.386844] Lustre: 269618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9651.996632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9652.002847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9652.168535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9656.821418] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9666.862536] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9671.432658] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9671.739625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9676.453168] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9676.637914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9682.916430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.925334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9682.926989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9682.951905] LustreError: Skipped 1 previous similar message [ 9684.682513] Lustre: server umount lustre-OST0000 complete [ 9688.545982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9693.152675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.155430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9693.168147] Lustre: Skipped 2 previous similar messages [ 9693.171421] Lustre: Skipped 2 previous similar messages [ 9694.699018] Lustre: server umount lustre-MDT0000 complete [ 9698.274769] LustreError: 267660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.307538] LustreError: 267660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9698.666923] LustreError: 267641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800501 with bad export cookie 14229415203380895921 [ 9698.683676] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9699.068919] Lustre: server umount lustre-MDT0001 complete [ 9720.796172] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9731.098990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.565204] LustreError: 272553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9731.616536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.624407] Lustre: Skipped 2 previous similar messages [ 9736.431537] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9741.795275] LustreError: 272554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9741.805956] LustreError: 272554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9744.680399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.954599] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9752.812808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9760.324380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9767.263544] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9776.373333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.588790] Lustre: lustre-OST0001: new disk, initializing [ 9776.596303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9776.609320] Lustre: 274861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9778.410797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9778.424662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9778.516120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9783.306982] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9794.927732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9799.737940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9805.136945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9805.320973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9811.941430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9811.954670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.970545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.976762] Lustre: Skipped 1 previous similar message [ 9826.271195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9826.868849] Lustre: server umount lustre-MDT0000 complete [ 9829.864512] LustreError: 272554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.902536] LustreError: 272554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9830.331898] LustreError: 272533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800633 with bad export cookie 14229415203380896803 [ 9830.343677] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9830.345425] LustreError: 272533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9830.635859] Lustre: server umount lustre-MDT0001 complete [ 9844.263873] Lustre: server umount lustre-OST0000 complete [ 9859.047427] Lustre: server umount lustre-OST0001 complete [ 9880.839553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9893.661100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9894.065037] LustreError: 277920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.102436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.110930] Lustre: Skipped 3 previous similar messages [ 9900.055434] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9909.068862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.125545] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9917.004665] Lustre: 279059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9924.237710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9930.239410] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9938.297409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.597943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9939.610517] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9944.568385] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9952.403906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9956.331717] Lustre: 280934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9962.017349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9962.099204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9971.730178] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:19:33 (1787800773) [ 9991.140261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9991.143099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9991.144448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9991.144455] Lustre: Skipped 12 previous similar messages [ 9991.173459] Lustre: Skipped 5 previous similar messages [10001.375818] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10001.746520] Lustre: server umount lustre-MDT0000 complete [10005.266153] LustreError: 277903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800808 with bad export cookie 14229415203380903348 [10005.290342] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.495942] LustreError: 277916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.515418] LustreError: 277916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10011.782327] Lustre: server umount lustre-MDT0001 complete [10021.911364] Lustre: server umount lustre-OST0000 complete [10026.467734] Lustre: server umount lustre-OST0001 complete [10035.447067] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10047.089596] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10101.605666] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10111.908258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.253890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10112.313507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10112.428393] Lustre: lustre-MDT0000: new disk, initializing [10112.553501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10118.152489] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10129.454371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.569271] Lustre: 285178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10129.893066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10135.294825] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10146.745441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10146.959895] Lustre: 286811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10148.378866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10153.482957] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10164.153547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.381973] Lustre: 287683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10164.501362] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10164.510223] Lustre: Skipped 6 previous similar messages [10166.165340] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10166.175841] Lustre: Skipped 1 previous similar message [10166.300551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10170.807274] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10181.445899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10185.399658] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10185.405157] Lustre: Skipped 1 previous similar message [10200.971499] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:23:23 (1787801003) [10206.690626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10206.712396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.728724] Lustre: Skipped 3 previous similar messages [10206.740528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10206.753040] Lustre: Skipped 15 previous similar messages [10209.192314] Lustre: server umount lustre-MDT0000 complete [10212.324164] LustreError: 285190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.345147] LustreError: 285190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10212.624297] LustreError: 285982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801015 with bad export cookie 14229415203380910985 [10212.629244] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10212.650119] LustreError: 285982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10213.104673] Lustre: server umount lustre-MDT0001 complete [10227.784942] Lustre: server umount lustre-OST0000 complete [10243.188069] Lustre: server umount lustre-OST0001 complete [10253.289530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.212592] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10266.865334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.372785] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10279.248478] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10285.016418] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10292.119438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10299.368989] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10306.526272] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10323.265601] loop1: detected capacity change from 0 to 204800000 [10323.626463] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10323.640990] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10323.653838] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10329.671473] loop1: detected capacity change from 0 to 204800000 [10335.590370] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.640778] LustreError: 289916:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5231: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10335.650893] LustreError: 289916:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5231 [10335.659560] LustreError: 289916:0:(mgs_handler.c:517:mgs_target_reg()) fs5231-MDT0001: Error checking target index: rc = -13 [10335.672222] LustreError: fs5231-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10335.680872] LustreError: 293645:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10335.687716] LustreError: 293645:0:(tgt_mount.c:2129:server_put_super()) no obd fs5231-MDT0001 [10335.692051] LustreError: 293645:0:(tgt_mount.c:116:server_deregister_mount()) fs5231-MDT0001 not registered [10335.726973] Lustre: server umount fs5231-MDT0001 complete [10335.729700] LustreError: 293645:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10346.926864] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10347.190971] Lustre: Setting parameter fs5231-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5231-MDT0001 [10347.285788] Lustre: srv-fs5231-MDT0001: No data found on store. Initialize space. [10347.293841] Lustre: Skipped 4 previous similar messages [10347.407897] Lustre: fs5231-MDT0001: new disk, initializing [10347.412716] Lustre: Skipped 3 previous similar messages [10347.627364] Lustre: srv-fs5231-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10347.634284] Lustre: 294112:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5231-MDT0001-mdtlov: can't create new llog: rc = -115 [10347.651317] LustreError: 294112:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5231-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10354.194189] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10361.861332] Lustre: server umount fs5231-MDT0001 complete [10375.696518] loop1: detected capacity change from 0 to 204800000 [10380.776089] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10382.428142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.822810] Lustre: 295680:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5231-MDT0001-mdtlov: can't create new llog: rc = -115 [10382.841551] LustreError: 295680:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5231-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10387.704240] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10392.637660] Lustre: server umount fs5231-MDT0001 complete [10404.773364] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [10404.778159] Lustre: Skipped 4 previous similar messages [10405.344355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10405.349837] LustreError: Skipped 1 previous similar message [10405.357557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10405.380829] Lustre: Skipped 3 previous similar messages [10408.950888] Lustre: server umount lustre-MDT0000 complete [10418.313856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.413716] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10423.665126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10423.821636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10430.848232] loop1: detected capacity change from 0 to 204800000 [10435.791284] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.278412] Lustre: 298127:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5231-MDT0001-mdtlov: can't create new llog: rc = -115 [10436.296849] LustreError: 298127:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5231-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10440.332845] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10445.099558] Lustre: server umount fs5231-MDT0001 complete [10467.808351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10469.872928] LustreError: 291761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.883399] LustreError: 291761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 109 previous similar messages [10471.654956] Lustre: server umount lustre-OST0000 complete [10481.787796] Lustre: server umount lustre-MDT0000 complete [10485.386935] LustreError: 290673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801288 with bad export cookie 14229415203380913666 [10485.395527] LustreError: 290673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10485.397226] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10485.693720] Lustre: server umount lustre-MDT0001 complete [10497.009882] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [10499.821927] Key type lgssc unregistered [10500.063826] LNet: 300477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10500.073046] LNetError: 300477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10500.098848] LNet: Removed LNI 192.168.201.148@tcp [10500.860519] Key type .llcrypt unregistered [10500.864836] Key type ._llcrypt unregistered [10521.370042] Key type ._llcrypt registered [10521.373828] Key type .llcrypt registered [10521.516718] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10523.304820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10523.318578] alg: No test for adler32 (adler32-zlib) [10524.546175] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [10524.813992] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [10526.471174] Key type lgssc registered [10527.988618] Lustre: Echo OBD driver; http://www.lustre.org/ [10536.829300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10536.861783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.575046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10543.690729] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10548.708544] LustreError: 301750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.735246] LustreError: 301750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10552.196931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.659160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10557.497589] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10565.027258] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10571.104806] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10577.079881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.328871] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.459419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10582.507332] LustreError: 303581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10583.935272] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10588.653035] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.678239] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10591.274204] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10593.774051] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.786178] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10604.013599] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.042261] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10607.826765] loop1: detected capacity change from 0 to 204800000 [10608.002163] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10608.024576] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10608.051251] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10612.300765] loop1: detected capacity change from 0 to 204800000 [10617.866411] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10617.979595] Lustre: fs15265-OST0000: new disk, initializing [10617.982236] Lustre: srv-fs15265-OST0000: No data found on store. Initialize space. [10617.986666] Lustre: 305484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10618.042362] Lustre: fs15265-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10624.490082] LustreError: 304712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10624.505382] LustreError: 304712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10624.764867] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10633.448083] Lustre: server umount fs15265-OST0000 complete [10646.307874] loop1: detected capacity change from 0 to 204800000 [10651.658340] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10652.026717] Lustre: fs15265-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.755494] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10660.325753] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10660.341315] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10660.364573] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10665.453047] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10668.085315] loop2: detected capacity change from 0 to 204800000 [10668.434669] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10668.444602] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10668.458759] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10673.666259] loop2: detected capacity change from 0 to 204800000 [10680.454825] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10680.543370] LustreError: fs15265-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10680.557459] LustreError: fs15265-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10680.564207] LustreError: 308525:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10680.572875] LustreError: 308525:0:(tgt_mount.c:2129:server_put_super()) no obd fs15265-OST0001 [10680.578376] LustreError: 308525:0:(tgt_mount.c:116:server_deregister_mount()) fs15265-OST0001 not registered [10680.640869] Lustre: server umount fs15265-OST0001 complete [10680.648742] LustreError: 308525:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10694.458486] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.556726] Lustre: fs15265-OST0001: new disk, initializing [10694.560485] Lustre: srv-fs15265-OST0001: No data found on store. Initialize space. [10694.566465] Lustre: 309077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10694.644483] Lustre: fs15265-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10701.856633] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10709.845446] Lustre: server umount fs15265-OST0000 complete [10718.401570] Lustre: server umount fs15265-OST0001 complete [10726.882905] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10726.901606] LustreError: 303876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [10730.588205] loop1: detected capacity change from 0 to 204800000 [10737.011680] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10737.278845] Lustre: fs15265-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10744.936661] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10751.369236] loop2: detected capacity change from 0 to 204800000 [10757.062000] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.273727] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10771.543301] Lustre: server umount fs15265-OST0000 complete [10780.394566] Lustre: server umount fs15265-OST0001 complete [10793.448263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10793.454195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10793.472467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10793.523169] Lustre: Skipped 1 previous similar message [10798.568862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.579613] Lustre: Skipped 2 previous similar messages [10803.690799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.700344] Lustre: Skipped 1 previous similar message [10807.263272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10807.506937] Lustre: server umount lustre-OST0000 complete [10813.922316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.922397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.952057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.958510] Lustre: Skipped 1 previous similar message [10817.343967] Lustre: server umount lustre-MDT0000 complete [10820.977735] LustreError: 301731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801624 with bad export cookie 8369724959120109491 [10820.985257] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.990110] LustreError: 301731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10821.319441] Lustre: server umount lustre-MDT0001 complete [10831.676484] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [10834.126514] Key type lgssc unregistered [10834.376666] LNet: 315002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10834.381547] LNetError: 315002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10834.395698] LNet: Removed LNI 192.168.201.148@tcp [10835.164163] Key type .llcrypt unregistered [10835.165672] Key type ._llcrypt unregistered [10863.259073] Key type ._llcrypt registered [10863.262387] Key type .llcrypt registered [10863.380796] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10875.820501] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10876.383352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10876.422893] alg: No test for adler32 (adler32-zlib) [10877.379387] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [10877.616823] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [10879.287976] Key type lgssc registered [10880.501814] Lustre: Echo OBD driver; http://www.lustre.org/ [10911.898388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10911.927668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.205150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10913.237455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10913.312801] Lustre: lustre-MDT0000: new disk, initializing [10913.376104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10913.391479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10917.144137] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10927.010735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.179403] Lustre: 318837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10927.226679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10927.230089] Lustre: Skipped 1 previous similar message [10927.312821] Lustre: lustre-MDT0001: new disk, initializing [10927.475039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10927.573748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10927.580217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10932.395186] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10940.654523] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10946.013802] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10951.921609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10952.197787] Lustre: lustre-OST0000: new disk, initializing [10952.211910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10952.217771] Lustre: 320805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10952.296461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10958.304284] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10959.944366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10959.975509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10960.105404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10967.953463] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10970.140910] Lustre: Mounted lustre-client - version 2.17.57_50_g504224c [10970.998606] Lustre: Unmounted lustre-client [10980.324405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10980.344023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10985.444298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10985.458549] Lustre: Skipped 1 previous similar message [10986.465298] Lustre: server umount lustre-OST0000 complete [10994.144127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10994.152504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.164499] Lustre: Skipped 1 previous similar message [10994.171119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10994.179898] Lustre: Skipped 1 previous similar message [10995.684234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10997.061149] Lustre: server umount lustre-MDT0000 complete [11000.367765] LustreError: 321669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801803 with bad export cookie 10344016694242491675 [11000.373563] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.379104] LustreError: 321669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11000.701963] Lustre: server umount lustre-MDT0001 complete [11010.231359] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11013.156484] Key type lgssc unregistered [11021.023240] LNet: 1 local NIs in recovery (showing 1): 192.168.201.148@tcp [11021.599776] LNet: 322968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11021.607259] LNetError: 322968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11101.481429] LNet: Removed LNI 192.168.201.148@tcp [11102.376580] Key type .llcrypt unregistered [11102.378407] Key type ._llcrypt unregistered [11118.824734] Key type ._llcrypt registered [11118.827711] Key type .llcrypt registered [11118.962343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11133.323690] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11134.485957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11134.522437] alg: No test for adler32 (adler32-zlib) [11135.598323] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11135.805518] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11137.472298] Key type lgssc registered [11138.777757] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.206889] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 23:39:31 (1787801971) [11176.005517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11176.078677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.343191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11177.385529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11177.522806] Lustre: lustre-MDT0000: new disk, initializing [11177.614904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11177.631081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11182.366936] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11192.727507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.959028] Lustre: 326190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11193.011627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11193.015372] Lustre: Skipped 1 previous similar message [11193.186918] Lustre: lustre-MDT0001: new disk, initializing [11193.540566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11193.662960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11193.688196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11199.184042] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11213.354753] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11220.142388] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11228.201140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.589134] Lustre: lustre-OST0000: new disk, initializing [11228.592822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11228.597788] Lustre: 328159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11228.755803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.937372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11234.952189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11235.149501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11235.232397] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11243.696943] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11260.897935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11260.910316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11260.918255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11260.930089] Lustre: Skipped 1 previous similar message [11260.946656] Lustre: Skipped 1 previous similar message [11264.058489] Lustre: server umount lustre-OST0000 complete [11270.113516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11270.119770] LustreError: Skipped 1 previous similar message [11270.121672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.127691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.136926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.146811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.162427] Lustre: Skipped 1 previous similar message [11273.779844] Lustre: server umount lustre-MDT0000 complete [11277.677502] LustreError: 326185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802080 with bad export cookie 8752984095710569238 [11277.692103] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11277.697230] LustreError: 326185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11278.021061] Lustre: server umount lustre-MDT0001 complete [11289.204403] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11292.013368] Key type lgssc unregistered [11292.294354] LNet: 330070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11292.308208] LNetError: 330070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11293.350886] LNet: Removed LNI 192.168.201.148@tcp [11294.249424] Key type .llcrypt unregistered [11294.251797] Key type ._llcrypt unregistered [11313.699333] Key type ._llcrypt registered [11313.702512] Key type .llcrypt registered [11313.838095] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11315.271671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11315.318920] alg: No test for adler32 (adler32-zlib) [11316.466452] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11316.777067] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11318.490531] Key type lgssc registered [11319.272468] Lustre: Echo OBD driver; http://www.lustre.org/ [11326.375507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11326.396313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.778472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11331.764269] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11337.189479] LustreError: 331345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.215401] LustreError: 331345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11339.495744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.850456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11344.528176] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11352.082468] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11357.500988] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11364.615718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.000198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.858246] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11378.283555] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11392.284518] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11394.052343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11400.168024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11400.171288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11400.179236] Lustre: Skipped 2 previous similar messages [11400.188222] Lustre: Skipped 2 previous similar messages [11401.661859] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [11404.307600] Lustre: server umount lustre-MDT0000 complete [11405.281480] LustreError: 331340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.314023] LustreError: 331340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11406.694456] LustreError: 331341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.528971] LustreError: 331326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802211 with bad export cookie 3291558843687662309 [11408.537218] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11408.548570] LustreError: 331326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11410.400391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.403065] LustreError: 331340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11410.420341] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11413.410865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11413.417602] Lustre: Skipped 1 previous similar message [11415.112278] Lustre: server umount lustre-MDT0001 complete [11419.376900] Lustre: server umount lustre-OST0000 complete [11428.959819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.043226] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11436.749831] Lustre: 335525:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.148@tcp [11438.867397] Lustre: server umount MGS complete [11446.439827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.782155] LustreError: 336097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.808860] LustreError: 336097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11446.849520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11451.112423] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11455.983785] LustreError: 336098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.032824] LustreError: 336098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11459.251208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.623041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11464.447157] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11473.152198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11473.544373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11479.339471] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11486.398950] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11487.788681] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11494.356653] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11499.620082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11505.634236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11505.646304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.660512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.693220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11507.701620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11509.933852] Lustre: server umount lustre-OST0000 complete [11515.874743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11515.881788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.894335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.898190] Lustre: Skipped 2 previous similar messages [11520.242113] Lustre: server umount lustre-MDT0000 complete [11523.041939] LustreError: 336092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.081579] LustreError: 336092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11525.141187] LustreError: 336079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802328 with bad export cookie 3291558843687663149 [11525.173031] LustreError: 336079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11525.175988] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.877577] Lustre: server umount lustre-MDT0001 complete [11538.075849] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11540.494305] Key type lgssc unregistered [11540.724053] LNet: 339539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11540.731397] LNetError: 339539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11540.761754] LNet: Removed LNI 192.168.201.148@tcp [11541.924823] Key type .llcrypt unregistered [11541.930237] Key type ._llcrypt unregistered [11564.469449] Key type ._llcrypt registered [11564.471758] Key type .llcrypt registered [11564.627786] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11580.669510] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11582.297810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11582.345019] alg: No test for adler32 (adler32-zlib) [11583.531751] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11583.888406] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11585.591537] Key type lgssc registered [11587.429629] Lustre: Echo OBD driver; http://www.lustre.org/ [11626.399558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11626.425148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.806613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11627.849789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11628.033586] Lustre: lustre-MDT0000: new disk, initializing [11628.159559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11628.204960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11632.218931] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11642.895925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.025545] Lustre: 342702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11643.070415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11643.073084] Lustre: Skipped 1 previous similar message [11643.135281] Lustre: lustre-MDT0001: new disk, initializing [11643.277831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11643.337528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11643.344412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11647.711991] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11658.072315] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11664.549112] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11671.329361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.870344] Lustre: lustre-OST0000: new disk, initializing [11671.874762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11671.885639] Lustre: 344671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11672.024366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11678.647440] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11680.356794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11680.379467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11680.593283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11689.423264] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11695.185659] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11695.416220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11701.139604] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11701.407898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11706.338786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11706.349272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.358432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11706.367725] Lustre: Skipped 1 previous similar message [11706.381478] Lustre: Skipped 1 previous similar message [11709.628776] Lustre: server umount lustre-OST0000 complete [11715.041232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11715.051086] LustreError: Skipped 1 previous similar message [11715.058772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.088223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.577802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.585355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11719.838709] Lustre: server umount lustre-MDT0000 complete [11723.397132] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802526 with bad export cookie 18241785707564632048 [11723.403155] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11723.409724] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11723.721985] Lustre: server umount lustre-MDT0001 complete [11732.776713] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:48:55 (1787802535) [11751.158685] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11763.390986] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11802.653865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.180223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11803.248972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11803.428852] Lustre: lustre-MDT0000: new disk, initializing [11803.560761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.586575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11808.399633] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11821.368920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.666266] Lustre: 349431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11821.764248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11821.773570] Lustre: Skipped 1 previous similar message [11821.911226] Lustre: lustre-MDT0001: new disk, initializing [11822.140207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11822.232956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11822.249133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11827.921276] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11838.973493] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11844.480228] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11851.831659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11852.155309] Lustre: lustre-OST0000: new disk, initializing [11852.157757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11852.161994] Lustre: 351401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11852.283821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11858.544606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11858.569799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11858.807571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11859.391276] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11871.709379] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11915.235460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.237292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11915.245184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11915.245192] Lustre: Skipped 1 previous similar message [11915.275361] Lustre: Skipped 1 previous similar message [11915.324844] LustreError: Skipped 1 previous similar message [11919.980622] Lustre: server umount lustre-OST0000 complete [11928.319598] Lustre: server umount lustre-MDT0000 complete [11929.569124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11929.585647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.618312] LustreError: 349443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.630065] LustreError: 349443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11930.597577] LustreError: 349439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.623469] LustreError: 349439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11933.847408] LustreError: 350228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802736 with bad export cookie 18241785707564632930 [11933.849184] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11933.861619] LustreError: 350228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11934.444534] Lustre: server umount lustre-MDT0001 complete [11949.665559] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11953.181962] Key type lgssc unregistered [11953.545765] LNet: 353583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11953.558697] LNetError: 353583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11953.569728] LNet: Removed LNI 192.168.201.148@tcp [11954.524665] Key type .llcrypt unregistered [11954.529630] Key type ._llcrypt unregistered [12052.583636] Key type ._llcrypt registered [12052.589923] Key type .llcrypt registered [12052.771532] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12053.259980] Key type .llcrypt unregistered [12053.261568] Key type ._llcrypt unregistered [12134.581448] Key type ._llcrypt registered [12134.583582] Key type .llcrypt registered [12134.688314] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12134.999421] Key type .llcrypt unregistered [12135.001386] Key type ._llcrypt unregistered [12164.673831] Key type ._llcrypt registered [12164.676536] Key type .llcrypt registered [12164.814914] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12165.083318] Key type .llcrypt unregistered [12165.087599] Key type ._llcrypt unregistered [12191.174929] Key type ._llcrypt registered [12191.178262] Key type .llcrypt registered [12191.266908] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12191.550493] Key type .llcrypt unregistered [12191.554861] Key type ._llcrypt unregistered [12271.755470] Key type ._llcrypt registered [12271.759118] Key type .llcrypt registered [12271.890726] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12272.220802] Key type .llcrypt unregistered [12272.225359] Key type ._llcrypt unregistered [12304.766642] Key type ._llcrypt registered [12304.768358] Key type .llcrypt registered [12304.859362] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12305.260352] Key type .llcrypt unregistered [12305.263879] Key type ._llcrypt unregistered [12336.142340] Key type ._llcrypt registered [12336.151132] Key type .llcrypt registered [12336.272174] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12336.601718] Key type .llcrypt unregistered [12336.606962] Key type ._llcrypt unregistered [12369.107542] Key type ._llcrypt registered [12369.110787] Key type .llcrypt registered [12369.206300] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12369.430488] Key type .llcrypt unregistered [12369.433295] Key type ._llcrypt unregistered [12397.696732] Key type ._llcrypt registered [12397.701919] Key type .llcrypt registered [12397.810479] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12398.049543] Key type .llcrypt unregistered [12398.056152] Key type ._llcrypt unregistered [12425.848835] Key type ._llcrypt registered [12425.850668] Key type .llcrypt registered [12425.942550] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12426.245400] Key type .llcrypt unregistered [12426.250441] Key type ._llcrypt unregistered [12493.877819] Key type ._llcrypt registered [12493.883651] Key type .llcrypt registered [12494.000499] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12494.405382] Key type .llcrypt unregistered [12494.406990] Key type ._llcrypt unregistered [12526.795597] Key type ._llcrypt registered [12526.797194] Key type .llcrypt registered [12526.900334] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12527.223980] Key type .llcrypt unregistered [12527.233651] Key type ._llcrypt unregistered [12556.083604] Key type ._llcrypt registered [12556.085871] Key type .llcrypt registered [12556.170904] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12556.439422] Key type .llcrypt unregistered [12556.442300] Key type ._llcrypt unregistered [12610.956069] Key type ._llcrypt registered [12610.960719] Key type .llcrypt registered [12611.171946] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12629.733343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12631.016530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12631.219246] alg: No test for adler32 (adler32-zlib) [12632.379946] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12632.582853] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [12634.296096] Key type lgssc registered [12636.055967] Lustre: Echo OBD driver; http://www.lustre.org/ [12681.938098] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12681.970078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.303193] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12683.339620] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12683.526916] Lustre: tst250fs-MDT0000: new disk, initializing [12683.648453] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.678882] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12689.905377] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12701.777955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.992652] Lustre: 373965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12702.146584] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12702.164592] Lustre: Skipped 1 previous similar message [12702.324167] Lustre: tst250fs-MDT0001: new disk, initializing [12702.528101] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12702.583377] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12702.601837] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12708.850318] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12716.437553] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12728.662830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12729.522138] Lustre: tst250fs-OST0000: new disk, initializing [12729.548863] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12729.579959] Lustre: 375598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12729.804316] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12734.045304] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12734.077178] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12734.287782] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12740.367344] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12750.980441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12751.152313] Lustre: tst250fs-OST0001: new disk, initializing [12751.155572] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12751.160760] Lustre: 376470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12751.257869] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12757.294377] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12758.133987] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12758.148043] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12758.265793] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12763.103917] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12765.805191] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12765.808968] Lustre: Skipped 2 previous similar messages [12789.233960] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.237958] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12789.245489] Lustre: Skipped 3 previous similar messages [12789.249965] Lustre: Skipped 2 previous similar messages [12793.335176] Lustre: server umount tst250fs-MDT0000 complete [12797.770291] LustreError: 373958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803600 with bad export cookie 13649161445170524633 [12797.780278] LustreError: 373958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12797.780971] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12798.181464] Lustre: server umount tst250fs-MDT0001 complete [12812.976157] Lustre: server umount tst250fs-OST0000 complete [12815.851371] Lustre: 372188:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787803602/real 1787803602] req@ffff8aed70050f00 x1874647774730880/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787803618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12815.895166] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12817.987979] Lustre: server umount tst250fs-OST0001 complete [12835.366167] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12843.592390] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12875.489825] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:07:57 (1787803677) [12876.792484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12892.258077] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12902.309952] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12940.136589] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12717 sec ========= 00:09:02 (1787803742) [12941.790929] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:09:04 (1787803744) === [12945.633319] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:09:08 (1787803748) === [12992.740285] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12995.266572] Key type lgssc unregistered [12995.607728] LNet: 385603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12995.616768] LNetError: 385603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12995.644240] LNet: Removed LNI 192.168.201.148@tcp [12996.669174] Key type .llcrypt unregistered [12996.676220] Key type ._llcrypt unregistered