[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483836502 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002259] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007422] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008043] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013089] *** VALIDATE selinux *** [ 0.021469] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025782] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030082] *** VALIDATE tmpfs *** [ 0.031472] *** VALIDATE proc *** [ 0.033088] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042213] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.044161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.047959] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052288] rcu: Hierarchical SRCU implementation. [ 0.054467] smp: Bringing up secondary CPUs ... [ 0.055619] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059197] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.219710] node 0 deferred pages initialised in 156ms [ 0.223008] devtmpfs: initialized [ 0.224198] x86/mm: Memory block size: 128MB [ 0.226805] gcov: version magic: 0x41383552 [ 0.230297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233316] pinctrl core: initialized pinctrl subsystem [ 0.236201] [ 0.236717] ************************************************************* [ 0.239014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241011] ** ** [ 0.242013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245013] ** ** [ 0.246012] ** This means that this kernel is built to expose internal ** [ 0.248013] ** IOMMU data structures, which may compromise security on ** [ 0.250016] ** your system. ** [ 0.253012] ** ** [ 0.255010] ** If you see this message and you are not debugging the ** [ 0.257012] ** kernel, report this immediately to your vendor! ** [ 0.259016] ** ** [ 0.262021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264013] ************************************************************* [ 0.267270] NET: Registered protocol family 16 [ 0.268379] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.271062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276479] cpuidle: using governor menu [ 0.278424] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281442] PCI: Using configuration type 1 for base access [ 0.283123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.297021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302048] cryptd: max_cpu_qlen set to 1000 [ 0.304327] ACPI: Added _OSI(Module Device) [ 0.306012] ACPI: Added _OSI(Processor Device) [ 0.308015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.315036] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.321554] ACPI: Interpreter enabled [ 0.323065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324014] ACPI: Using IOAPIC for interrupt routing [ 0.326099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330298] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.340711] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.343042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.346022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354370] acpiphp: Slot [2] registered [ 0.355111] acpiphp: Slot [5] registered [ 0.356040] acpiphp: Slot [6] registered [ 0.357197] acpiphp: Slot [7] registered [ 0.359148] acpiphp: Slot [8] registered [ 0.360135] acpiphp: Slot [9] registered [ 0.362080] acpiphp: Slot [10] registered [ 0.363060] acpiphp: Slot [3] registered [ 0.364128] acpiphp: Slot [4] registered [ 0.366104] acpiphp: Slot [11] registered [ 0.367106] acpiphp: Slot [12] registered [ 0.368082] acpiphp: Slot [13] registered [ 0.369087] acpiphp: Slot [14] registered [ 0.370084] acpiphp: Slot [15] registered [ 0.372101] acpiphp: Slot [16] registered [ 0.373080] acpiphp: Slot [17] registered [ 0.374089] acpiphp: Slot [18] registered [ 0.375108] acpiphp: Slot [19] registered [ 0.377128] acpiphp: Slot [20] registered [ 0.379146] acpiphp: Slot [21] registered [ 0.380084] acpiphp: Slot [22] registered [ 0.381118] acpiphp: Slot [23] registered [ 0.382071] acpiphp: Slot [24] registered [ 0.382978] acpiphp: Slot [25] registered [ 0.384068] acpiphp: Slot [26] registered [ 0.385024] acpiphp: Slot [27] registered [ 0.385826] acpiphp: Slot [28] registered [ 0.387121] acpiphp: Slot [29] registered [ 0.387984] acpiphp: Slot [30] registered [ 0.389169] acpiphp: Slot [31] registered [ 0.390060] PCI host bridge to bus 0000:00 [ 0.391015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393071] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.396016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.400132] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.402689] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404987] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413584] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417389] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.422014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.424291] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.426485] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.428041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430637] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.449015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.454022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458450] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.497019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.506460] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.512018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.521019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549727] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.557013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.577039] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.591017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.601607] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.609028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.616044] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.629029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.639177] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.650021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.690021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703934] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.711028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.717030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.729016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.737000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.738454] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.741389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.745314] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.747356] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.752157] iommu: Default domain type: Passthrough [ 0.754548] SCSI subsystem initialized [ 0.755141] ACPI: bus type USB registered [ 0.757137] usbcore: registered new interface driver usbfs [ 0.759087] usbcore: registered new interface driver hub [ 0.761097] usbcore: registered new device driver usb [ 0.762182] pps_core: LinuxPPS API ver. 1 registered [ 0.764014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.767065] PTP clock support registered [ 0.769127] EDAC MC: Ver: 3.0.0 [ 0.771153] PCI: Using ACPI for IRQ routing [ 0.772833] NetLabel: Initializing [ 0.774013] NetLabel: domain hash size = 128 [ 0.776010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.777084] NetLabel: unlabeled traffic allowed by default [ 0.780106] vgaarb: loaded [ 0.781260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.790000] clocksource: Switched to clocksource kvm-clock [ 0.901926] VFS: Disk quotas dquot_6.6.0 [ 0.903259] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.905737] *** VALIDATE ramfs *** [ 0.906910] *** VALIDATE hugetlbfs *** [ 0.908499] pnp: PnP ACPI init [ 0.910988] pnp: PnP ACPI: found 6 devices [ 0.929744] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933176] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935531] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.937851] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.940355] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942805] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.945708] NET: Registered protocol family 2 [ 0.948217] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.953123] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.956669] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961398] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964876] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968026] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971173] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974282] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977352] NET: Registered protocol family 1 [ 0.979887] RPC: Registered named UNIX socket transport module. [ 0.982199] RPC: Registered udp transport module. [ 0.984033] RPC: Registered tcp transport module. [ 0.985436] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.987863] NET: Registered protocol family 44 [ 0.989734] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.991978] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994210] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.996642] PCI: CLS 0 bytes, default 64 [ 0.998347] Unpacking initramfs... [ 2.392788] debug: unmapping init [mem 0xffff8999fcc54000-0xffff8999fffbffff] [ 2.396765] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.398776] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.401508] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.900640] Initialise system trusted keyrings [ 2.902424] Key type blacklist registered [ 2.904326] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.913470] zbud: loaded [ 2.916356] *** VALIDATE nfs *** [ 2.917550] *** VALIDATE nfs4 *** [ 2.918937] pstore: using deflate compression [ 2.922662] Platform Keyring initialized [ 3.014332] NET: Registered protocol family 38 [ 3.016153] Key type asymmetric registered [ 3.018693] Asymmetric key parser 'x509' registered [ 3.021564] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.025933] io scheduler mq-deadline registered [ 3.028363] io scheduler kyber registered [ 3.030453] io scheduler bfq registered [ 3.032982] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.036502] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.039495] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.042449] ACPI: Power Button [PWRF] [ 3.047283] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.054793] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.075234] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.084389] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.104888] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.133685] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.164174] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.168650] Non-volatile memory driver v1.3 [ 3.170390] Linux agpgart interface v0.103 [ 3.200424] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.203229] vda: detected capacity change from 0 to 74862592 [ 3.219164] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.222141] vdb: detected capacity change from 0 to 1073741824 [ 3.236446] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.239364] vdc: detected capacity change from 0 to 2621440000 [ 3.255529] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.258552] vdd: detected capacity change from 0 to 2621440000 [ 3.283746] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.286848] vde: detected capacity change from 0 to 4294967296 [ 3.309957] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.313198] vdf: detected capacity change from 0 to 4294967296 [ 3.323630] libphy: Fixed MDIO Bus: probed [ 3.330747] usbcore: registered new interface driver usbserial_generic [ 3.334530] usbserial: USB Serial support registered for generic [ 3.337984] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.343369] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.346103] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.348710] mousedev: PS/2 mouse device common for all mice [ 3.352451] rtc_cmos 00:05: RTC can wake from S4 [ 3.356143] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.360527] rtc_cmos 00:05: registered as rtc0 [ 3.362228] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.363804] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.368071] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.369629] intel_pstate: CPU model not supported [ 3.374991] hid: raw HID events driver (C) Jiri Kosina [ 3.376616] usbcore: registered new interface driver usbhid [ 3.378101] usbhid: USB HID core driver [ 3.379353] drop_monitor: Initializing network drop monitor service [ 3.381274] Initializing XFRM netlink socket [ 3.382803] NET: Registered protocol family 10 [ 3.385164] Segment Routing with IPv6 [ 3.386217] NET: Registered protocol family 17 [ 3.387907] mpls_gso: MPLS GSO support [ 3.393942] RAS: Correctable Errors collector initialized. [ 3.396093] AVX version of gcm_enc/dec engaged. [ 3.397717] AES CTR mode by8 optimization enabled [ 3.457763] sched_clock: Marking stable (3457741946, 0)->(4346861323, -889119377) [ 3.461276] registered taskstats version 1 [ 3.463289] Loading compiled-in X.509 certificates [ 3.465282] zswap: loaded using pool lzo/zbud [ 3.484526] Key type big_key registered [ 3.496494] Key type encrypted registered [ 3.498416] ima: No TPM chip found, activating TPM-bypass! [ 3.500501] ima: Allocated hash algorithm: sha1 [ 3.502280] ima: No architecture policies found [ 3.503859] evm: Initialising EVM extended attributes: [ 3.505854] evm: security.selinux [ 3.507103] evm: security.ima [ 3.508140] evm: security.capability [ 3.509520] evm: HMAC attrs: 0x1 [ 3.511549] rtc_cmos 00:05: setting system clock to 2026-07-28 10:02:06 UTC (1785232926) [ 3.517626] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.521163] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 3.530091] Write protecting the kernel read-only data: 28672k [ 3.533249] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.535881] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 3.562631] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.571256] systemd[1]: Detected virtualization kvm. [ 3.573182] systemd[1]: Detected architecture x86-64. [ 3.574978] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.605761] systemd[1]: No hostname configured. [ 3.607765] systemd[1]: Set hostname to . [ 3.610299] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613569] systemd[1]: Initializing machine ID from random generator. [ 3.639544] random: ln: uninitialized urandom read (6 bytes read) [ 3.732933] random: systemd: uninitialized urandom read (16 bytes read) [ 3.735268] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.739876] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.746467] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.006621] device-mapper: uevent: version 1.0.3 [ 5.015760] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.542711] virtio_net virtio0 ens2: renamed from eth0 [ 6.703354] random: fast init done [ 6.984922] scsi host0: ata_piix [ 7.006837] scsi host1: ata_piix [ 7.014750] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.022469] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.019844] random: crng init done [ 12.028922] random: 7 urandom warning(s) missed due to ratelimiting [ 13.742885] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.416580] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.690769] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.585093] SELinux: Disabled at runtime. [ 20.717996] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.741817] systemd[1]: Detected virtualization kvm. [ 20.743454] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.388961] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.394811] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.406170] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.414133] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.422227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.434643] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.450543] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ 22.585747] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.999351] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.041922] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.138257] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.955931] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.007426] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.615435] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.359653] NFS: Registering the id_resolver key type [ 31.361117] Key type id_resolver registered [ 31.362727] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 68.361052] hrtimer: interrupt took 1152520 ns [ 104.345952] libcfs: loading out-of-tree module taints kernel. [ 104.428536] Key type ._llcrypt registered [ 104.434481] Key type .llcrypt registered [ 104.610294] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 126.854729] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 128.486171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.517234] alg: No test for adler32 (adler32-zlib) [ 130.147524] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 131.295897] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 133.049937] Key type lgssc registered [ 135.457748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.340273] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 199.206941] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 213.001593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.077755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.438566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.463661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.536633] Lustre: lustre-MDT0000: new disk, initializing [ 214.640224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.661059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.987741] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.872200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.055675] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.100976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 236.106397] Lustre: Skipped 1 previous similar message [ 236.273790] Lustre: lustre-MDT0001: new disk, initializing [ 236.376545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.404624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.420182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.879717] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.216722] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 258.073649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.390817] Lustre: lustre-OST0000: new disk, initializing [ 258.393587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.399183] Lustre: 8458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.472962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.916660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 261.927952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.014843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.960300] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.506995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 282.679495] Lustre: lustre-OST0001: new disk, initializing [ 282.687403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 282.696938] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 282.812793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 288.793770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 288.808427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 288.971106] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.942782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.123798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.732390] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.525568] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.957358] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 329.813789] Lustre: DEBUG MARKER: Client: 2.17.54.204 [ 332.852131] Lustre: DEBUG MARKER: MDS: 2.17.54.204 [ 335.913883] Lustre: DEBUG MARKER: OSS: 2.17.54.204 [ 337.676938] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 28 06:07:38 EDT 2026 [ 357.000647] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 358.685392] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 374.753239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 374.766501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.783152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.287939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.288397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.299544] Lustre: Skipped 2 previous similar messages [ 376.309264] Lustre: Skipped 3 previous similar messages [ 380.046149] Lustre: server umount lustre-MDT0000 complete [ 384.062700] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233307 with bad export cookie 1489242513334279923 [ 384.073371] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.528273] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.543321] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.544760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.561492] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 386.591698] Lustre: Skipped 1 previous similar message [ 389.535919] LustreError: 8632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.536142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.551262] LustreError: 8632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 389.586082] Lustre: Skipped 1 previous similar message [ 390.543293] Lustre: server umount lustre-MDT0001 complete [ 394.361843] Lustre: server umount lustre-OST0000 complete [ 397.922104] Lustre: server umount lustre-OST0001 complete [ 405.110280] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 415.665856] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 458.745598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.183213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.245158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.449113] Lustre: lustre-MDT0000: new disk, initializing [ 459.638669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.672560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 466.265728] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 479.129311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.221961] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 479.271703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 479.275188] Lustre: Skipped 1 previous similar message [ 479.350732] Lustre: lustre-MDT0001: new disk, initializing [ 479.448993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.514866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 479.529942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 483.607403] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 494.034330] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.664695] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.959076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 508.269495] Lustre: lustre-OST0000: new disk, initializing [ 508.271446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 508.277979] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 508.361490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.842187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 509.850812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 509.979919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 515.180271] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 524.138950] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.232990] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 528.445734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 532.564511] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 532.779546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 535.523239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.523803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.553371] Lustre: Skipped 1 previous similar message [ 535.568060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.656060] LustreError: 17040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.670240] LustreError: 17040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 540.850347] Lustre: server umount lustre-OST0000 complete [ 545.765154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.779298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.800463] Lustre: Skipped 2 previous similar messages [ 545.807041] Lustre: Skipped 1 previous similar message [ 551.033964] Lustre: server umount lustre-MDT0000 complete [ 555.182346] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233478 with bad export cookie 1489242513334281456 [ 555.184880] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.202058] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 555.511425] Lustre: server umount lustre-MDT0001 complete [ 561.902302] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:11:23 (1785233483) [ 581.392502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.939363] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.085596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.208589] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 587.233974] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.250184] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 596.450250] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.481811] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 596.584362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.510212] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 611.328737] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 617.822240] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 620.235210] Lustre: Failing over lustre-MDT0000 [ 620.478888] Lustre: server umount lustre-MDT0000 complete [ 621.286491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.302926] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.303782] Lustre: Skipped 1 previous similar message [ 621.329896] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 637.919335] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785233544/real 1785233544] req@ffff899a77e4a300 x1871952534991488/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785233560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 637.958371] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.875421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.398492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.401480] Lustre: Skipped 1 previous similar message [ 648.455967] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 652.659815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 653.802614] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 653.816420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 653.853799] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 661.569597] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 666.789691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.807523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.818032] Lustre: Skipped 4 previous similar messages [ 672.083826] Lustre: server umount lustre-MDT0000 complete [ 672.227753] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.253150] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 675.271530] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233598 with bad export cookie 1489242513334282870 [ 675.272893] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.282904] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 675.543184] Lustre: server umount lustre-MDT0001 complete [ 683.447694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.946918] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 697.847103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.190675] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 709.602347] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 715.544438] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 718.252323] Lustre: Failing over lustre-MDT0000 [ 718.674589] Lustre: server umount lustre-MDT0000 complete [ 718.816351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.835709] Lustre: Skipped 1 previous similar message [ 738.603697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.733814] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.042111] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 744.119106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 744.123549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 744.148838] Lustre: Skipped 1 previous similar message [ 744.176755] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 745.656596] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 756.466399] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 763.873490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.147630] Lustre: server umount lustre-MDT0000 complete [ 769.002047] LustreError: 23269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.033880] LustreError: 23269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 771.071884] LustreError: 23251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233694 with bad export cookie 1489242513334283808 [ 771.083128] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.089444] LustreError: 23251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 771.731948] Lustre: server umount lustre-MDT0001 complete [ 781.336838] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:15:02 (1785233702) [ 797.728369] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 808.007168] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 844.460426] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 855.452305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.797169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 855.835136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 855.975379] Lustre: lustre-MDT0000: new disk, initializing [ 856.098351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.111323] Lustre: Skipped 3 previous similar messages [ 856.138061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 861.194731] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 871.387664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.509659] Lustre: 29641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 871.751992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 876.507539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 886.063920] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.058658] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 900.192729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.531742] Lustre: 31611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 902.239270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 902.329068] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 902.383066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 902.415106] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 907.358624] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 917.895383] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.903473] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 948.712824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.721890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.723870] LustreError: Skipped 1 previous similar message [ 948.728386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.728394] Lustre: Skipped 1 previous similar message [ 948.769688] Lustre: Skipped 4 previous similar messages [ 954.220317] Lustre: server umount lustre-OST0000 complete [ 964.066232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.304694] Lustre: server umount lustre-MDT0000 complete [ 969.188120] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.208104] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 969.281942] LustreError: 31609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233892 with bad export cookie 1489242513334284214 [ 969.290778] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.297790] LustreError: 31609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 969.815072] Lustre: server umount lustre-MDT0001 complete [ 981.012454] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 984.234595] Key type lgssc unregistered [ 984.579041] LNet: 33588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.591526] LNetError: 33588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.636779] LNet: Removed LNI 192.168.202.153@tcp [ 986.610595] Key type .llcrypt unregistered [ 986.615996] Key type ._llcrypt unregistered [ 1017.252718] Key type ._llcrypt registered [ 1017.257184] Key type .llcrypt registered [ 1017.379245] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1019.186072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.208532] alg: No test for adler32 (adler32-zlib) [ 1020.513854] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1020.756466] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1022.399333] Key type lgssc registered [ 1023.765404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.925725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.961133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.443120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.636388] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1043.941261] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.967392] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1047.941612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.371459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.382114] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1063.834849] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.506524] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.985876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.560846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.493482] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1093.908126] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1095.932161] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.230120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1121.789659] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:20:42 (1785234042) [ 1123.872516] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1125.661756] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1132.506691] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:20:53 (1785234053) [ 1142.428893] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:21:03 (1785234063) [ 1143.517703] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1152.547478] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1152.554863] Lustre: Skipped 2 previous similar messages [ 1154.769275] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:21:15 (1785234075) [ 1157.054459] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1157.060213] Lustre: Skipped 2 previous similar messages [ 1165.305079] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1165.313162] Lustre: Skipped 1 previous similar message [ 1182.929248] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1182.939757] Lustre: Skipped 5 previous similar messages [ 1185.906883] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:21:46 (1785234106) [ 1190.093350] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1192.735753] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1204.078933] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:22:04 (1785234124) [ 1219.835291] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1219.858664] Lustre: Skipped 7 previous similar messages [ 1222.534784] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:22:23 (1785234143) [ 1241.570548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.585112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.598631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.617723] Lustre: Skipped 1 previous similar message [ 1246.689282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.700880] Lustre: Skipped 1 previous similar message [ 1251.822479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.837720] Lustre: Skipped 1 previous similar message [ 1253.343225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1253.644903] Lustre: server umount lustre-OST0000 complete [ 1258.472148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1258.485494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.501501] Lustre: Skipped 1 previous similar message [ 1258.508712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.054801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.173118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.179370] Lustre: Skipped 3 previous similar messages [ 1272.288537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1272.605592] Lustre: server umount lustre-MDT0000 complete [ 1276.763239] LustreError: 34984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234199 with bad export cookie 13030398637364054767 [ 1276.773893] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.779760] LustreError: 34984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1277.295110] Lustre: server umount lustre-MDT0001 complete [ 1289.561064] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1292.905394] Key type lgssc unregistered [ 1293.192460] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.199442] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.214938] LNet: Removed LNI 192.168.202.153@tcp [ 1294.350846] Key type .llcrypt unregistered [ 1294.354408] Key type ._llcrypt unregistered [ 1316.743549] Key type ._llcrypt registered [ 1316.754237] Key type .llcrypt registered [ 1317.002243] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1335.514877] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1337.198792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.243536] alg: No test for adler32 (adler32-zlib) [ 1338.487970] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1338.848564] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1340.551217] Key type lgssc registered [ 1341.916554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.766526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.806448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.146511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1383.181189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1383.287556] Lustre: lustre-MDT0000: new disk, initializing [ 1383.463193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.478863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1388.607796] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1401.232598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.378994] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1401.426897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1401.430454] Lustre: Skipped 1 previous similar message [ 1401.498741] Lustre: lustre-MDT0001: new disk, initializing [ 1401.595416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.632775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1401.643110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1406.774397] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1416.747947] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1423.436830] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.173900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.509910] Lustre: lustre-OST0000: new disk, initializing [ 1430.515309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1430.520409] Lustre: 48009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1430.644548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.486041] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1439.323461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1439.339143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1439.442158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1447.689620] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.891619] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1453.373517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1458.611916] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1458.846462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1465.323752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1465.324315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.338056] LustreError: Skipped 1 previous similar message [ 1465.338735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1465.373740] Lustre: Skipped 1 previous similar message [ 1467.468187] Lustre: server umount lustre-OST0000 complete [ 1473.511678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1473.518401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.527765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.531857] Lustre: Skipped 1 previous similar message [ 1475.553098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.553586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.577085] Lustre: Skipped 1 previous similar message [ 1479.001436] Lustre: server umount lustre-MDT0000 complete [ 1483.060285] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234406 with bad export cookie 15568905658280749752 [ 1483.064228] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1483.069205] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1483.283882] Lustre: server umount lustre-MDT0001 complete [ 1495.342476] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:26:55 (1785234415) [ 1501.027589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.491987] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.504555] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1501.544396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.031289] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1506.788519] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.905850] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.697150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.999822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1520.885474] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1529.389434] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1535.868679] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1544.090892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1544.648126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.157697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1562.653852] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.785429] LustreError: 53000:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1568.826963] Lustre: Modifying parameter general.timeout=1 in log params [ 1569.970532] Lustre: Modifying parameter general.timeout=2 in log params [ 1572.126412] Lustre: Modifying parameter general.timeout=4 in log params [ 1572.130655] Lustre: Skipped 1 previous similar message [ 1576.522290] Lustre: Modifying parameter general.timeout=9 in log params [ 1576.525672] Lustre: Skipped 4 previous similar messages [ 1585.315209] Lustre: Modifying parameter general.timeout=17 in log params [ 1585.323669] Lustre: Skipped 7 previous similar messages [ 1601.803479] Lustre: Modifying parameter general.timeout=34 in log params [ 1601.809962] Lustre: Skipped 16 previous similar messages [ 1634.589076] Lustre: Modifying parameter general.timeout=67 in log params [ 1634.603423] Lustre: Skipped 32 previous similar messages [ 1699.313381] Lustre: Modifying parameter general.timeout=125 in log params [ 1699.318175] Lustre: Skipped 57 previous similar messages [ 1717.142479] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:30:37 (1785234637) [ 1720.352289] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1722.244145] Lustre: 60065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1724.294572] Lustre: 60163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1728.461346] Lustre: 60361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1728.477914] Lustre: 60361:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1763.016454] Lustre: 61836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1763.029056] Lustre: 61836:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1763.054087] LustreError: 61836:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1768.370324] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:31:29 (1785234689) [ 1786.338702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.342774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.373266] Lustre: Skipped 2 previous similar messages [ 1786.404426] Lustre: Skipped 2 previous similar messages [ 1791.460476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.491060] Lustre: Skipped 2 previous similar messages [ 1792.040429] Lustre: server umount lustre-MDT0000 complete [ 1796.578818] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.594447] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1797.457481] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234720 with bad export cookie 15568905658280750641 [ 1797.476894] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.335328] Lustre: server umount lustre-MDT0001 complete [ 1814.246152] Lustre: server umount lustre-OST0000 complete [ 1848.098126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.518203] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1848.591834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1848.605015] Lustre: Skipped 27 previous similar messages [ 1848.887919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.809537] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1866.243311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.374837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1866.409864] Lustre: 64374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1873.369494] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1883.392569] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.461422] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.344587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.709463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1899.158190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.170858] Lustre: Skipped 1 previous similar message [ 1907.773964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1917.001671] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.146966] Lustre: 66942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1928.672443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1928.676671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.693216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.146112] Lustre: server umount lustre-OST0000 complete [ 1938.401847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1938.421438] LustreError: Skipped 1 previous similar message [ 1938.428092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.449349] Lustre: Skipped 1 previous similar message [ 1943.811359] Lustre: server umount lustre-MDT0000 complete [ 1944.545373] LustreError: 65485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.570660] LustreError: 65485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1948.173071] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234871 with bad export cookie 15568905658280753896 [ 1948.175568] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.183281] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1948.718639] Lustre: server umount lustre-MDT0001 complete [ 1961.155921] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1964.432275] Key type lgssc unregistered [ 1964.677088] LNet: 68153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.683684] LNetError: 68153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.697276] LNet: Removed LNI 192.168.202.153@tcp [ 1965.733680] Key type .llcrypt unregistered [ 1965.735147] Key type ._llcrypt unregistered [ 1994.643367] Key type ._llcrypt registered [ 1994.645646] Key type .llcrypt registered [ 1994.732564] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1996.466802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.545526] alg: No test for adler32 (adler32-zlib) [ 1997.722949] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1998.030807] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1999.807385] Key type lgssc registered [ 2000.979697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.829100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.861096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.318840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.490849] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2022.891544] LustreError: 69571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.915332] LustreError: 69571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2028.000703] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.522620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.422595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.574456] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2044.193483] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.565148] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2059.416483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.989992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.234887] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2076.000112] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2089.829747] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2103.266515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.271265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.280690] Lustre: Skipped 2 previous similar messages [ 2103.282915] Lustre: Skipped 1 previous similar message [ 2108.385270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.404684] Lustre: Skipped 3 previous similar messages [ 2108.753384] Lustre: server umount lustre-MDT0000 complete [ 2113.507195] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.544433] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2113.634751] LustreError: 69551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235036 with bad export cookie 3982707083348035428 [ 2113.639729] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.645759] LustreError: 69551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2113.980858] Lustre: server umount lustre-MDT0001 complete [ 2128.503549] Lustre: server umount lustre-OST0000 complete [ 2158.923479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.457576] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2159.526237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2159.873642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.665636] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2175.824534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.934804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2175.966436] Lustre: 74446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2176.511258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.957877] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2191.938871] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.752110] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2207.536103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.856637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2208.449226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.836176] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2225.099614] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.100386] Lustre: 77026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2229.318388] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2239.971932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2239.994543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.018061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.038924] Lustre: Skipped 1 previous similar message [ 2241.803707] Lustre: server umount lustre-OST0000 complete [ 2248.161542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2248.173489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.192571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.198524] Lustre: Skipped 1 previous similar message [ 2250.217616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.295711] Lustre: server umount lustre-MDT0000 complete [ 2255.328580] LustreError: 74457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.354971] LustreError: 74457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2256.800753] LustreError: 74437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235179 with bad export cookie 3982707083348036275 [ 2256.807735] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.814619] LustreError: 74437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2257.180341] Lustre: server umount lustre-MDT0001 complete [ 2270.127822] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2273.429364] Key type lgssc unregistered [ 2273.755798] LNet: 78238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.759383] LNetError: 78238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.790876] LNet: Removed LNI 192.168.202.153@tcp [ 2275.747758] Key type .llcrypt unregistered [ 2275.751243] Key type ._llcrypt unregistered [ 2300.848486] Key type ._llcrypt registered [ 2300.856296] Key type .llcrypt registered [ 2301.004854] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2302.065292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.362466] alg: No test for adler32 (adler32-zlib) [ 2303.481660] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2303.697537] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2305.359590] Key type lgssc registered [ 2306.893331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2316.281696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2316.323443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.985314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.680907] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2327.589851] LustreError: 79651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.607923] LustreError: 79651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2330.393758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.323442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2335.283552] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2342.964981] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2351.114033] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2359.562358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2360.272426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.075699] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2376.089632] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2383.719636] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2384.700949] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2385.839280] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2388.614165] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2388.620414] Lustre: Skipped 2 previous similar messages [ 2393.554716] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2393.558617] Lustre: Skipped 4 previous similar messages [ 2402.339861] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2402.351391] Lustre: Skipped 8 previous similar messages [ 2418.932475] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2418.948668] Lustre: Skipped 15 previous similar messages [ 2451.240699] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2451.245405] Lustre: Skipped 29 previous similar messages [ 2497.344521] Lustre: 87763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2500.363830] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:43:41 (1785235421) [ 2502.284731] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2504.405601] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:43:45 (1785235425) [ 2508.775085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.779356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2508.795706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2508.815790] Lustre: Skipped 1 previous similar message [ 2513.919456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2513.933392] Lustre: Skipped 1 previous similar message [ 2514.972155] Lustre: server umount lustre-OST0000 complete [ 2520.548885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.556259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.575232] Lustre: Skipped 1 previous similar message [ 2520.584824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.144561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.146989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.248678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.252762] Lustre: Skipped 2 previous similar messages [ 2534.368945] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2534.509836] Lustre: server umount lustre-MDT0000 complete [ 2538.435236] LustreError: 79633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235461 with bad export cookie 16568183674342647649 [ 2538.442715] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.450827] LustreError: 79633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2538.821706] Lustre: server umount lustre-MDT0001 complete [ 2551.140412] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2553.513294] Key type lgssc unregistered [ 2553.736287] LNet: 89014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.742653] LNetError: 89014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.761118] LNet: Removed LNI 192.168.202.153@tcp [ 2554.685355] Key type .llcrypt unregistered [ 2554.687844] Key type ._llcrypt unregistered [ 2558.359138] Key type ._llcrypt registered [ 2558.366446] Key type .llcrypt registered [ 2558.646603] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2565.300590] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules [ 2567.625540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2567.672345] alg: No test for adler32 (adler32-zlib) [ 2568.956382] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2569.392612] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2571.161281] Key type lgssc registered [ 2573.109586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2574.773951] LustreError: 90265:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2596.856697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2598.365377] LustreError: 90265:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2608.024241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2608.035704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.639964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.826347] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2619.874450] LustreError: 91291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.901826] LustreError: 91291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2622.695450] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:45:43 (1785235543) [ 2624.995947] LustreError: 91290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.524315] LustreError: 91323:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2627.532455] LustreError: 91323:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2627.539936] LustreError: 91323:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2627.831751] Lustre: server umount lustre-MDT0000 complete [ 2645.867803] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2649.634075] Key type lgssc unregistered [ 2650.100839] LNet: 92815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2650.110920] LNetError: 92815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.140853] LNet: Removed LNI 192.168.202.153@tcp [ 2651.522262] Key type .llcrypt unregistered [ 2651.523590] Key type ._llcrypt unregistered [ 2671.613593] Key type ._llcrypt registered [ 2671.615704] Key type .llcrypt registered [ 2671.745359] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2672.623405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.662617] alg: No test for adler32 (adler32-zlib) [ 2673.978476] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2674.300076] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2676.063273] Key type lgssc registered [ 2677.343821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2686.479833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2686.523154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.262424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.813675] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2698.722708] LustreError: 94036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.745314] LustreError: 94036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2703.848332] LustreError: 94035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.128233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.847794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2710.630391] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2719.002246] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2724.827558] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.978815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2732.655728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.710647] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2749.215891] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2802.477551] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:48:43 (1785235723) [ 2804.966433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2807.466195] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:48:47 (1785235727) [ 2815.971910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.989837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.992790] Lustre: Skipped 2 previous similar messages [ 2816.018394] Lustre: Skipped 2 previous similar messages [ 2821.091619] LustreError: 94035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.116993] LustreError: 94035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2821.143201] Lustre: server umount lustre-MDT0000 complete [ 2826.005324] LustreError: 94017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235748 with bad export cookie 3734657610413963045 [ 2826.024704] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.028345] LustreError: 94017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2826.211587] LustreError: 94032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.232818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.278676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2826.629917] Lustre: server umount lustre-MDT0001 complete [ 2834.211491] Lustre: server umount lustre-OST0000 complete [ 2846.168884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.699253] LustreError: 98331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.879596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.312721] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2856.418425] LustreError: 98332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.453885] LustreError: 98332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2862.337094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.442439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.055443] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2877.865905] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.615590] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2899.231812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.512247] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2899.525165] LustreError: 98318:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2899.565500] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2899.576862] LustreError: 100414:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2899.591543] LustreError: 100414:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2899.597120] LustreError: 100414:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2899.731889] Lustre: server umount lustre-OST0000 complete [ 2899.738310] LustreError: 100414:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2910.070815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.426343] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2910.436812] LustreError: 98320:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2910.460632] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2910.471524] LustreError: 100881:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2910.483266] LustreError: 100881:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2910.493633] LustreError: 100881:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2910.568584] Lustre: server umount lustre-OST0000 complete [ 2910.575267] LustreError: 100881:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2921.653574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.858716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2921.866198] Lustre: Found index 0 for lustre-OST0000, updating log [ 2921.873154] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2922.005727] Lustre: lustre-OST0000: new disk, initializing [ 2922.013888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2922.022442] Lustre: 101419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2922.334323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.935814] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2934.609987] Lustre: Failing over lustre-OST0000 [ 2934.781704] Lustre: server umount lustre-OST0000 complete [ 2939.873484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2939.881859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.891968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.949633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.951465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.894075] Lustre: server umount lustre-MDT0000 complete [ 2948.068958] LustreError: 98328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.088411] LustreError: 98328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2950.055215] LustreError: 100433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235873 with bad export cookie 3734657610413965782 [ 2950.075472] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.616788] Lustre: server umount lustre-MDT0001 complete [ 2970.925782] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 06:51:32 (1785235892) [ 2990.204813] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3001.938292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.394938] LustreError: 103765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.600263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.231649] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3018.792768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.325775] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3029.388288] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3037.087721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.364741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.366898] Lustre: Skipped 1 previous similar message [ 3038.391219] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3044.531982] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3044.578202] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3044.581917] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3045.698755] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3055.586529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.711389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3055.784657] Lustre: lustre-OST0001: new disk, initializing [ 3055.788224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3055.795247] Lustre: 106083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3063.902852] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3064.440526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3064.455638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3064.564069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3072.742993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3083.124046] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3090.407132] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.411680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3090.426336] Lustre: Skipped 2 previous similar messages [ 3090.441592] Lustre: Skipped 3 previous similar messages [ 3100.642684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3100.656064] Lustre: Skipped 5 previous similar messages [ 3102.175144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3102.386439] Lustre: server umount lustre-MDT0001 complete [ 3102.843566] LustreError: 106563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.867861] LustreError: 106563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3113.707884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.807241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3113.809843] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3113.863960] Lustre: 103753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3114.485473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3114.490349] Lustre: Skipped 1 previous similar message [ 3115.921031] LustreError: 107663:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3115.936336] LustreError: 107663:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3115.946398] LustreError: 103758:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3116.031992] LustreError: 107668:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -17 [ 3116.038742] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3116.038742] [ 3119.630199] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3119.829229] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3131.739791] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 06:54:12 (1785236052) [ 3154.974286] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3177.389529] Lustre: 109700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3201.896876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3206.101313] Lustre: 110836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3303.398739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3303.404159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.413631] LustreError: 103765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.427494] LustreError: 103765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3303.540923] Lustre: server umount lustre-MDT0000 complete [ 3308.539112] LustreError: 103747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785236231 with bad export cookie 3734657610413966783 [ 3308.563105] LustreError: 103747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3308.573633] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.193167] Lustre: server umount lustre-MDT0001 complete [ 3316.256731] Lustre: server umount lustre-OST0000 complete [ 3320.051572] Lustre: server umount lustre-OST0001 complete [ 3327.785047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3339.076712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3349.428635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3360.024934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3387.160616] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3399.069904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.087327] Lustre: lustre-MDT0000: reset Object Index mappings [ 3399.831912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.900902] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3415.029255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3415.064796] Lustre: lustre-MDT0001: reset Object Index mappings [ 3421.904949] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3425.673424] Lustre: 115936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3433.580207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.378521] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3440.593956] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3444.471153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3450.546326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.523238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3456.550684] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3458.749223] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3467.157806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3471.916364] Lustre: 117814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3529.398001] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:00:50 (1785236450) [ 3558.887730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3558.888361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.910805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.932828] Lustre: Skipped 7 previous similar messages [ 3558.942953] Lustre: Skipped 5 previous similar messages [ 3561.842815] Lustre: server umount lustre-MDT0000 complete [ 3564.015337] LustreError: 116315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.065455] LustreError: 116315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 3566.682375] LustreError: 114769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785236489 with bad export cookie 3734657610414013641 [ 3566.685858] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3566.693328] LustreError: 114769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3573.482566] Lustre: server umount lustre-MDT0001 complete [ 3584.610802] Lustre: server umount lustre-OST0000 complete [ 3595.185702] Lustre: server umount lustre-OST0001 complete [ 3603.136740] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3614.900971] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3656.979658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.207784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3657.232714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3657.322114] Lustre: lustre-MDT0000: new disk, initializing [ 3657.472569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.477859] Lustre: Skipped 3 previous similar messages [ 3657.515865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3663.088549] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3670.642530] Lustre: server umount lustre-MDT0000 complete [ 3681.048439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.470888] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3681.485475] Lustre: 122109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3681.499675] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3686.128441] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3691.382637] Lustre: server umount lustre-MDT0000 complete [ 3701.423921] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:03:42 (1785236622) [ 3703.438799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3706.103618] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:03:46 (1785236626) [ 3727.647524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3739.613615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.133893] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3754.844291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.057855] Lustre: 124162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3755.173465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3755.179471] Lustre: Skipped 1 previous similar message [ 3755.353860] Lustre: lustre-MDT0001: new disk, initializing [ 3755.596662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3755.612602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3760.881780] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3766.703268] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3774.073106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3774.417081] Lustre: lustre-OST0000: new disk, initializing [ 3774.420312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3774.428268] Lustre: 125737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3775.869610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3775.884896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3775.950052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3781.540958] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3793.091769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.245055] Lustre: lustre-OST0001: new disk, initializing [ 3793.247790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3793.252326] Lustre: 126609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3794.755414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3794.762652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3794.881324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3799.491435] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3810.352612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3814.580493] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4254.170822] Lustre: 133034:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4254.178410] Lustre: 133034:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4254.185603] Lustre: 133034:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4254.189764] Lustre: 133034:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4254.196600] Lustre: 133034:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4254.201857] Lustre: 133034:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4557.295747] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:17:57 (1785237477) [ 4574.689814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4574.694696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.701601] Lustre: Skipped 3 previous similar messages [ 4574.709985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4574.712306] Lustre: Skipped 4 previous similar messages [ 4578.958863] Lustre: server umount lustre-MDT0000 complete [ 4581.381209] LustreError: 127387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.401746] LustreError: 127387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4583.252223] LustreError: 124154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785237506 with bad export cookie 3734657610414083060 [ 4583.263962] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.274141] LustreError: 124154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4586.465134] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.478403] Lustre: Skipped 3 previous similar messages [ 4586.484560] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4586.490046] Lustre: Skipped 5 previous similar messages [ 4589.844477] Lustre: server umount lustre-MDT0001 complete [ 4600.071116] Lustre: server umount lustre-OST0000 complete [ 4609.636888] Lustre: server umount lustre-OST0001 complete [ 4617.267680] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4626.916676] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4668.715476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.104798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4669.145638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4669.245539] Lustre: lustre-MDT0000: new disk, initializing [ 4669.362448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.374969] Lustre: Skipped 5 previous similar messages [ 4669.415243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4674.614405] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4685.284417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.488518] Lustre: 136090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4685.546914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4685.550534] Lustre: Skipped 1 previous similar message [ 4685.607582] Lustre: lustre-MDT0001: new disk, initializing [ 4685.764596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4685.777288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4691.240776] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4701.554222] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4707.315547] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4715.056814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.391482] Lustre: lustre-OST0000: new disk, initializing [ 4715.397588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4715.409703] Lustre: 138061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4717.336189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4717.344902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4717.443243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4722.183153] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4731.431599] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4742.364542] Lustre: lustre-MDD0000: changelog on [ 4745.277799] Lustre: lustre-MDD0001: changelog on [ 6732.534404] Lustre: lustre-MDD0001: changelog off [ 6735.364175] Lustre: 136098:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6736.782200] Lustre: lustre-MDD0000: changelog off [ 6745.057305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.061332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.063227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.116416] Lustre: Skipped 2 previous similar messages [ 6745.121939] Lustre: Skipped 2 previous similar messages [ 6750.181447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.184836] Lustre: Skipped 2 previous similar messages [ 6750.439426] Lustre: server umount lustre-OST0000 complete [ 6759.398732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6759.410376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.438611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6763.084836] Lustre: server umount lustre-MDT0000 complete [ 6765.540631] LustreError: 136102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.571824] LustreError: 136102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6768.027097] LustreError: 136896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785239691 with bad export cookie 3734657610414901213 [ 6768.033512] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.042074] LustreError: 136896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6768.390606] Lustre: server umount lustre-MDT0001 complete [ 6783.183964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6786.728386] Key type lgssc unregistered [ 6787.018817] LNet: 147214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.025898] LNetError: 147214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6788.072028] LNet: Removed LNI 192.168.202.153@tcp [ 6789.213145] Key type .llcrypt unregistered [ 6789.214396] Key type ._llcrypt unregistered [ 6815.240603] Key type ._llcrypt registered [ 6815.251478] Key type .llcrypt registered [ 6815.403315] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 6833.293671] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6835.106437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6835.370443] alg: No test for adler32 (adler32-zlib) [ 6836.559189] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 6836.884321] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6838.617061] Key type lgssc registered [ 6840.089991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6886.282248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6886.320110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.701057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6887.737678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6887.943562] Lustre: lustre-MDT0000: new disk, initializing [ 6888.084852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.104885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6893.421428] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6905.260824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.370466] Lustre: 150417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6905.457335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6905.461479] Lustre: Skipped 1 previous similar message [ 6905.551264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6905.597836] Lustre: lustre-MDT0001: new disk, initializing [ 6905.698612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.740487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6905.764500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6910.922487] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6922.062695] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6929.160820] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6939.998903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.531670] Lustre: lustre-OST0000: new disk, initializing [ 6940.534685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6940.544152] Lustre: 152386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6940.625903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.465459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6948.481465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6948.696201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6949.371800] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6962.120164] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6968.551365] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_module obdecho/obdecho [ 6970.577986] LustreError: 153390:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6970.583661] LustreError: 153390:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6972.681664] LustreError: 153487:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6987.753970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6987.761844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.779179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.813989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.822884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.847483] Lustre: Skipped 1 previous similar message [ 6991.432881] Lustre: server umount lustre-MDT0000 complete [ 6995.069435] LustreError: 150411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785239918 with bad export cookie 538257073554738409 [ 6995.069779] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.075344] LustreError: 150411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6995.353697] Lustre: server umount lustre-MDT0001 complete [ 7009.443424] Lustre: server umount lustre-OST0000 complete [ 7039.344535] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 7052.775826] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7100.966940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.396825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7101.448270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7101.566739] Lustre: lustre-MDT0000: new disk, initializing [ 7101.683236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.721226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7107.498314] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7119.506684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.687469] Lustre: 157210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7119.747110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7119.750809] Lustre: Skipped 1 previous similar message [ 7119.831156] Lustre: lustre-MDT0001: new disk, initializing [ 7119.977398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.021155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7120.031899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7127.297160] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7140.819899] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7148.988891] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7158.255331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.533443] Lustre: lustre-OST0000: new disk, initializing [ 7158.536936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7158.544595] Lustre: 159191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7158.673232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7167.109287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7167.121894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7167.391979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7167.603689] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7180.422635] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7186.514532] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7186.912765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7193.202360] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7193.582470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7198.185231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7198.187432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.202272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.228965] Lustre: Skipped 1 previous similar message [ 7198.231295] Lustre: Skipped 2 previous similar messages [ 7202.950507] Lustre: server umount lustre-OST0000 complete [ 7208.420310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.425314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7208.437932] Lustre: Skipped 1 previous similar message [ 7208.447872] Lustre: Skipped 2 previous similar messages [ 7214.877652] Lustre: server umount lustre-MDT0000 complete [ 7218.661691] LustreError: 158577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.672951] LustreError: 158577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7219.503724] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240142 with bad export cookie 538257073554739368 [ 7219.510124] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.517368] LustreError: 157201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7219.930542] Lustre: server umount lustre-MDT0001 complete [ 7224.645678] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:02:25 (1785240145) [ 7226.660362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7229.151945] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:02:29 (1785240149) [ 7239.475454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.981699] LustreError: 161605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.118826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.307317] LustreError: 161606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.346599] LustreError: 161606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7248.845405] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7251.454438] LustreError: 161605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.558185] LustreError: 161606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.825090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.832358] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7281.256848] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7289.336391] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7299.483157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.154374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.156845] Lustre: Skipped 1 previous similar message [ 7309.011614] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7317.981544] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7327.207447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.221728] Lustre: Skipped 2 previous similar messages [ 7327.222501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7327.238601] Lustre: Skipped 3 previous similar messages [ 7330.159550] Lustre: server umount lustre-MDT0000 complete [ 7332.329611] LustreError: 163071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.343028] LustreError: 163071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7335.137991] LustreError: 161588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240258 with bad export cookie 538257073554740250 [ 7335.158879] LustreError: 161588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7335.164406] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.445305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.720793] Lustre: server umount lustre-MDT0001 complete [ 7350.149776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7366.386348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.705336] LustreError: 165453:0:(llog.c:1655:llog_backup()) MGC192.168.202.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7366.718331] Lustre: 165453:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7369.389293] LustreError: 165458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.405605] LustreError: 165458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7405.030623] LustreError: 165458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.064788] LustreError: 165458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 7427.552368] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7427.555116] Lustre: 165453:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7468.547336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7470.707817] LustreError: 165491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.750498] LustreError: 165491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7473.468314] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7473.659376] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7482.673398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.171931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7496.182031] LustreError: Skipped 1 previous similar message [ 7505.376642] LustreError: 148639:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f28cf291 x1871959565680000/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7510.173376] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7511.020772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7517.560257] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7519.153545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7525.471477] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7527.352842] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7546.884813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7546.890308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.904725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.926354] Lustre: Skipped 2 previous similar messages [ 7546.951622] Lustre: Skipped 6 previous similar messages [ 7551.108707] Lustre: server umount lustre-MDT0000 complete [ 7555.243910] LustreError: 164393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240478 with bad export cookie 538257073554746711 [ 7555.252977] LustreError: 164393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7555.254173] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.592715] Lustre: server umount lustre-MDT0001 complete [ 7570.822579] Lustre: server umount lustre-OST0000 complete [ 7595.471913] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 7606.341808] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7648.897287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.207718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7649.240906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7649.368976] Lustre: lustre-MDT0000: new disk, initializing [ 7649.506668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.520757] Lustre: Skipped 1 previous similar message [ 7649.554338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7655.737176] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7668.522469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.701792] Lustre: 170564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7669.115472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7674.664812] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7685.157354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.079331] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7700.545718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.925639] Lustre: lustre-OST0000: new disk, initializing [ 7700.932865] Lustre: Skipped 1 previous similar message [ 7700.942131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7700.948160] Lustre: Skipped 2 previous similar messages [ 7700.952630] Lustre: 172532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7708.239571] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7709.294944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7709.302582] Lustre: Skipped 1 previous similar message [ 7709.304837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7709.451139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7720.186674] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7725.498851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7725.792368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7731.779441] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7732.147383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7735.266168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.278094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.283877] LustreError: Skipped 1 previous similar message [ 7735.284565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.321121] Lustre: Skipped 1 previous similar message [ 7740.840847] Lustre: server umount lustre-OST0000 complete [ 7751.392649] Lustre: server umount lustre-MDT0000 complete [ 7754.799079] LustreError: 170556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240677 with bad export cookie 538257073554763546 [ 7754.807111] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7754.809653] LustreError: 170556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7755.082638] Lustre: server umount lustre-MDT0001 complete [ 7764.735643] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:11:26 (1785240686) [ 7771.640592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7772.051379] LustreError: 174941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.074171] LustreError: 174941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 33 previous similar messages [ 7777.158275] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7789.561513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.037675] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7806.203432] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7813.969819] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7823.273631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.730828] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7842.623289] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7854.056146] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:12:55 (1785240775) [ 7860.706346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7860.712287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.718716] Lustre: Skipped 2 previous similar messages [ 7860.724361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.730275] Lustre: Skipped 7 previous similar messages [ 7865.521080] Lustre: server umount lustre-OST0000 complete [ 7875.983161] Lustre: server umount lustre-MDT0000 complete [ 7880.171745] LustreError: 177632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240803 with bad export cookie 538257073554764421 [ 7880.178446] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.182503] LustreError: 177632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7880.691504] Lustre: server umount lustre-MDT0001 complete [ 7893.048493] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7896.712715] Key type lgssc unregistered [ 7897.025444] LNet: 178673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7897.035165] LNetError: 178673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.094896] LNet: Removed LNI 192.168.202.153@tcp [ 7899.223597] Key type .llcrypt unregistered [ 7899.230027] Key type ._llcrypt unregistered [ 7923.888881] Key type ._llcrypt registered [ 7923.890918] Key type .llcrypt registered [ 7924.012673] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7925.463478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7925.593886] alg: No test for adler32 (adler32-zlib) [ 7926.716647] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7926.998567] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7928.727161] Key type lgssc registered [ 7930.430762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7939.461848] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7939.508287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.167425] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7960.287464] LustreError: 179916:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 7965.407767] LustreError: 179916:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7965.432685] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7965.444696] LustreError: MGC192.168.202.153@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7965.484791] LustreError: 179916:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7965.513237] LustreError: 179916:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7965.530250] LustreError: 179916:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7965.692067] Lustre: server umount lustre-OST0000 complete [ 7965.699188] LustreError: 179916:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7979.624842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.511850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.693586] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7991.783208] LustreError: 180414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.811572] LustreError: 180414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7996.905845] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.901452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.567178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.786474] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8016.916731] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8023.934217] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8032.162266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.580512] LustreError: 182232:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8032.592410] Lustre: 182232:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.153@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8032.811301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.551147] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8048.775476] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8051.680721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8051.684132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.711139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.289965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.294102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.307127] Lustre: Skipped 2 previous similar messages [ 8058.026609] Lustre: server umount lustre-MDT0000 complete [ 8061.412180] LustreError: 180411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.454856] LustreError: 180411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8062.372245] LustreError: 181150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785240985 with bad export cookie 16412110316626494312 [ 8062.378149] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.387420] LustreError: 181150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8062.801128] Lustre: server umount lustre-MDT0001 complete [ 8078.201735] Lustre: server umount lustre-OST0000 complete [ 8089.775603] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8102.846846] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8145.915053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.351446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8146.418059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8146.610357] Lustre: lustre-MDT0000: new disk, initializing [ 8146.780064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.836863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8152.043153] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8164.078838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.281319] Lustre: 185762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8164.389164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8164.399676] Lustre: Skipped 1 previous similar message [ 8164.534070] Lustre: lustre-MDT0001: new disk, initializing [ 8164.669831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8164.707840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8164.722618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8170.565135] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8183.150770] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8189.533519] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8196.119135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.431742] Lustre: lustre-OST0000: new disk, initializing [ 8196.439261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8196.455362] Lustre: 187728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8196.552948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8202.297603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8202.306247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8202.382784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8203.503539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8215.042646] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8221.293182] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8221.476087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8227.370610] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8227.822222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8233.442367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8233.444404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.462577] LustreError: Skipped 1 previous similar message [ 8233.467207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8233.483687] Lustre: Skipped 2 previous similar messages [ 8233.499146] Lustre: Skipped 1 previous similar message [ 8236.473107] Lustre: server umount lustre-OST0000 complete [ 8241.632500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8241.647058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.673155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.092335] Lustre: server umount lustre-MDT0000 complete [ 8248.800484] LustreError: 186924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.819041] LustreError: 186924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8251.122634] LustreError: 185754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785241174 with bad export cookie 16412110316626495033 [ 8251.125066] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.134713] LustreError: 185754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8251.609716] Lustre: server umount lustre-MDT0001 complete [ 8263.284213] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:19:44 (1785241184) [ 8282.648301] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8285.930406] Key type lgssc unregistered [ 8286.337741] LNet: 190676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.344571] LNetError: 190676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8286.368852] LNet: Removed LNI 192.168.202.153@tcp [ 8287.395318] Key type .llcrypt unregistered [ 8287.397670] Key type ._llcrypt unregistered [ 8307.525254] Key type ._llcrypt registered [ 8307.527911] Key type .llcrypt registered [ 8307.652555] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8309.471985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8309.552105] alg: No test for adler32 (adler32-zlib) [ 8310.701894] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 8310.930886] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8312.698306] Key type lgssc registered [ 8314.122304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8322.990065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8323.029038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.641836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.776318] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8334.827022] LustreError: 191897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.847912] LustreError: 191897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8339.937610] LustreError: 191896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8341.700377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.299233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.734235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8356.504253] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8363.817508] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8373.697096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8374.031193] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8374.034557] LustreError: 193765:0:(llog.c:1696:llog_backup()) MGC192.168.202.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8374.046175] Lustre: 193765:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.153@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8374.287821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.913454] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8390.456554] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8393.698042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8393.707946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.723822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.307838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.322677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.237464] Lustre: server umount lustre-MDT0000 complete [ 8403.444560] LustreError: 191891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.478893] LustreError: 191891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8403.615953] LustreError: 191878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785241326 with bad export cookie 14929736707624669572 [ 8403.622750] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8403.633312] LustreError: 191878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8403.993930] Lustre: server umount lustre-MDT0001 complete [ 8418.748395] Lustre: server umount lustre-OST0000 complete [ 8428.887160] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8439.283987] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8484.690278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.059588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8485.088848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8485.214204] Lustre: lustre-MDT0000: new disk, initializing [ 8485.355370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.389353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8491.476594] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8504.978260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.102583] Lustre: 197292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8505.159404] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8505.162017] Lustre: Skipped 1 previous similar message [ 8505.166901] LustreError: 198075:0:(llog.c:1696:llog_backup()) MGC192.168.202.153@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8505.175065] LustreError: 198075:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8505.180478] Lustre: 198075:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.153@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8505.207843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8505.212212] Lustre: Skipped 1 previous similar message [ 8505.334701] Lustre: lustre-MDT0001: new disk, initializing [ 8505.498804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8505.533700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8505.558809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8511.106759] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8523.543085] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8530.648607] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8539.147674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.550580] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8539.552807] Lustre: Skipped 1 previous similar message [ 8539.555409] LustreError: 199239:0:(llog.c:1696:llog_backup()) MGC192.168.202.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8539.560679] LustreError: 199239:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8539.575624] Lustre: 199239:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.153@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8539.587656] Lustre: 199239:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8539.608429] Lustre: lustre-OST0000: new disk, initializing [ 8539.613785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8539.620277] Lustre: 199263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8539.733592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8545.402791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8545.412976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8545.570984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8547.724268] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8557.993142] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8562.637245] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8562.795668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8568.101313] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8568.313485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8571.365094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.384339] Lustre: Skipped 1 previous similar message [ 8571.387555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8571.391663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8571.399483] LustreError: Skipped 1 previous similar message [ 8571.411520] Lustre: Skipped 3 previous similar messages [ 8576.492754] LustreError: 200603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.512256] LustreError: 200603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8576.743948] Lustre: server umount lustre-OST0000 complete [ 8581.601840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.622330] Lustre: Skipped 1 previous similar message [ 8581.643474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.652757] Lustre: Skipped 1 previous similar message [ 8586.720809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.733873] Lustre: Skipped 1 previous similar message [ 8587.599969] Lustre: server umount lustre-MDT0000 complete [ 8590.848303] LustreError: 197284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785241513 with bad export cookie 14929736707624670286 [ 8590.855463] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8590.866544] LustreError: 197284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8591.386585] Lustre: server umount lustre-MDT0001 complete [ 8603.785932] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:25:23 (1785241523) [ 8630.835128] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8644.811594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.194147] LustreError: 202408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.228894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.857198] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8655.843628] LustreError: 202408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.881455] LustreError: 202408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8662.999000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.762729] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8674.500369] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8682.928693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8683.415134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.435670] Lustre: Skipped 1 previous similar message [ 8691.523342] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8703.447497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.783422] Lustre: lustre-OST0001: new disk, initializing [ 8703.796386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8703.805331] Lustre: 204717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8711.826329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8711.859602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8712.071128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8712.944893] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8726.464806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8735.449195] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8738.420765] Lustre: DEBUG MARKER: ADD OST3 [ 8744.043670] loop: module loaded [ 8744.212683] loop0: detected capacity change from 0 to 204800000 [ 8744.448979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8744.463788] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8744.489730] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8753.514337] Lustre: DEBUG MARKER: STOP OST3 [ 8753.613207] loop0: detected capacity change from 0 to 204800000 [ 8761.050733] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8761.309619] Lustre: lustre-OST0003: new disk, initializing [ 8761.320751] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8761.330359] Lustre: 206764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8761.467314] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8761.479874] Lustre: Skipped 1 previous similar message [ 8762.488113] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8762.814086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8762.832630] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8762.887274] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8764.511935] LustreError: 206855:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8764.644469] Lustre: Failing over lustre-OST0003 [ 8764.842492] Lustre: server umount lustre-OST0003 complete [ 8768.016389] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8768.020258] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8768.050283] Lustre: Skipped 1 previous similar message [ 8768.060494] LustreError: 203893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8768.089273] LustreError: 203893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8769.586400] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8783.610394] Lustre: DEBUG MARKER: START OST3 again [ 8792.028556] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.426900] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8793.649364] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8793.787841] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8793.788035] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8802.539276] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8827.979410] Lustre: Failing over lustre-OST0003 [ 8828.160530] Lustre: server umount lustre-OST0003 complete [ 8828.401572] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.410297] LustreError: 203896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.430537] Lustre: Skipped 2 previous similar messages [ 8828.479218] LustreError: 203896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8848.870249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.884525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.904758] Lustre: Skipped 2 previous similar messages [ 8848.923740] Lustre: Skipped 3 previous similar messages [ 8854.342738] Lustre: server umount lustre-MDT0000 complete [ 8858.739800] LustreError: 205484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785241781 with bad export cookie 14929736707624671168 [ 8858.753934] LustreError: 205484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8858.754300] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8865.421195] Lustre: server umount lustre-MDT0001 complete [ 8870.622772] Lustre: server umount lustre-OST0000 complete [ 8875.248647] Lustre: server umount lustre-OST0001 complete [ 8883.640539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8893.701493] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8938.658965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.954682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8938.996812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8939.057927] Lustre: lustre-MDT0000: new disk, initializing [ 8939.183113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.187629] Lustre: Skipped 1 previous similar message [ 8939.208215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8943.004483] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8955.652889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.857965] Lustre: 212335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8955.929298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8955.931143] Lustre: Skipped 1 previous similar message [ 8956.036363] Lustre: lustre-MDT0001: new disk, initializing [ 8956.156888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8956.168870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8961.294573] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8973.533453] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8982.831336] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8991.297973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8991.562936] Lustre: lustre-OST0000: new disk, initializing [ 8991.566469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8991.571522] Lustre: 214307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8992.697379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8992.704312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8992.824774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8999.586584] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9010.274561] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9015.001497] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9015.199867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9019.797905] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9020.003933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9023.460070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.462186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9023.472129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9023.472139] Lustre: Skipped 8 previous similar messages [ 9023.475802] Lustre: Skipped 4 previous similar messages [ 9023.537260] LustreError: Skipped 1 previous similar message [ 9036.255229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9036.535550] Lustre: server umount lustre-OST0000 complete [ 9043.424602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9047.080233] Lustre: server umount lustre-MDT0000 complete [ 9049.061445] LustreError: 212346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.081926] LustreError: 212346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 9051.675404] LustreError: 212329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785241974 with bad export cookie 14929736707624673464 [ 9051.677279] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9051.697634] LustreError: 212329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9052.291403] Lustre: server umount lustre-MDT0001 complete [ 9058.696766] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:32:58 (1785241978) [ 9084.881075] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9100.043723] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9149.272389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.839901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9149.968195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9150.223444] Lustre: lustre-MDT0000: new disk, initializing [ 9150.518370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9157.284939] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9171.608767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.723538] Lustre: 218944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9171.920113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9175.798555] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9187.513614] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9195.600518] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9203.519146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.894553] Lustre: 220914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9204.049619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.052451] Lustre: Skipped 4 previous similar messages [ 9205.243346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9205.411161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9210.502307] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9220.663083] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9226.334829] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9226.707323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9233.605923] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9233.950455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9236.465827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9236.485051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.500295] Lustre: Skipped 2 previous similar messages [ 9236.507425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9236.509981] Lustre: Skipped 7 previous similar messages [ 9250.273680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9250.632437] Lustre: server umount lustre-OST0000 complete [ 9270.239257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9270.718915] Lustre: server umount lustre-MDT0000 complete [ 9271.781767] LustreError: 218949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.811213] LustreError: 218949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9276.169639] LustreError: 219738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785242199 with bad export cookie 14929736707624674346 [ 9276.170888] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.194561] LustreError: 219738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9276.661317] Lustre: server umount lustre-MDT0001 complete [ 9285.794596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.017793] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9301.841158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.144090] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9317.636701] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9327.160550] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9337.104644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9346.191158] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9356.301701] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9369.575915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9369.589544] LustreError: Skipped 1 previous similar message [ 9369.596648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.622914] Lustre: Skipped 3 previous similar messages [ 9369.628734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.633648] Lustre: Skipped 10 previous similar messages [ 9373.707637] Lustre: server umount lustre-OST0000 complete [ 9392.095216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9392.379885] Lustre: server umount lustre-MDT0000 complete [ 9396.892850] LustreError: 223114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785242319 with bad export cookie 14929736707624675221 [ 9396.897714] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9396.905037] LustreError: 223114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9397.364866] Lustre: server umount lustre-MDT0001 complete [ 9496.584165] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9500.162153] Key type lgssc unregistered [ 9500.538959] LNet: 226802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9500.548962] LNetError: 226802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9501.608585] LNet: Removed LNI 192.168.202.153@tcp [ 9502.756159] Key type .llcrypt unregistered [ 9502.761123] Key type ._llcrypt unregistered [ 9532.660444] Key type ._llcrypt registered [ 9532.662582] Key type .llcrypt registered [ 9532.771103] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9552.407267] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9553.638476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9553.785467] alg: No test for adler32 (adler32-zlib) [ 9555.105243] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 9555.347096] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9557.015263] Key type lgssc registered [ 9558.231553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9603.293000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9603.323947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9604.694364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9604.737341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9604.850934] Lustre: lustre-MDT0000: new disk, initializing [ 9604.928140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9604.945355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.192596] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9623.533257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9623.697628] Lustre: 230157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9623.779301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9623.786913] Lustre: Skipped 1 previous similar message [ 9623.912767] Lustre: lustre-MDT0001: new disk, initializing [ 9624.174190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9624.318140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9624.344547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9629.946581] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9642.230511] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9651.832684] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9660.657082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9660.936900] Lustre: lustre-OST0000: new disk, initializing [ 9660.939790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9660.944641] Lustre: 232128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9661.018607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.639053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9666.643751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9666.789918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9667.218450] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9678.250661] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9684.477047] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9684.889218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9690.226898] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9690.451956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9697.764503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.773407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9697.773930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9697.776160] LustreError: Skipped 1 previous similar message [ 9699.103944] Lustre: server umount lustre-OST0000 complete [ 9708.035861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.044896] Lustre: Skipped 2 previous similar messages [ 9708.049781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9708.052071] Lustre: Skipped 1 previous similar message [ 9713.327819] Lustre: server umount lustre-MDT0000 complete [ 9720.445076] LustreError: 230151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785242643 with bad export cookie 3078514825577002176 [ 9720.454216] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9720.822972] Lustre: server umount lustre-MDT0001 complete [ 9746.146452] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9757.285847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.586442] LustreError: 235064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.605317] LustreError: 235064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9757.619173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.786600] LustreError: 235063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.627628] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9767.908697] LustreError: 235064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9773.042162] LustreError: 235063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.019425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.557500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9781.519196] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9785.453555] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9797.907177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9798.633989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.204870] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9820.168291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9820.434349] Lustre: lustre-OST0001: new disk, initializing [ 9820.437696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9820.447483] Lustre: 237375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9820.550976] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9827.540191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9827.549540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9827.736746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9830.035353] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9843.465400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9851.787212] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9868.111602] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:46:28 (1785242788) [ 9873.891435] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.894942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9873.901140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9873.933077] Lustre: Skipped 3 previous similar messages [ 9873.972839] Lustre: Skipped 4 previous similar messages [ 9876.930446] Lustre: server umount lustre-MDT0000 complete [ 9879.021651] LustreError: 236519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.052498] LustreError: 236519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9881.067249] LustreError: 235043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785242804 with bad export cookie 3078514825577003051 [ 9881.071956] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9881.077179] LustreError: 235043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9881.676992] Lustre: server umount lustre-MDT0001 complete [ 9896.176972] Lustre: server umount lustre-OST0000 complete [ 9910.894236] Lustre: server umount lustre-OST0001 complete [ 9920.482692] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9931.263083] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9983.808657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9984.286291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9984.333021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9984.467696] Lustre: lustre-MDT0000: new disk, initializing [ 9984.561542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.610981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9989.873034] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10005.298778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10005.478101] Lustre: 241328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10005.557723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10005.563349] Lustre: Skipped 1 previous similar message [10005.633077] Lustre: lustre-MDT0001: new disk, initializing [10005.718572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10005.727775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10011.936454] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10025.432983] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10034.634682] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10044.867487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10045.268700] Lustre: lustre-OST0000: new disk, initializing [10045.276826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10045.286342] Lustre: 243299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10047.082754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10047.112250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10047.235500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10053.354549] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10064.614537] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.926344] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10071.154177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10076.433198] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10076.742076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10083.300481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10083.300931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10083.305379] LustreError: Skipped 1 previous similar message [10083.326660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10083.346340] Lustre: Skipped 1 previous similar message [10085.258799] Lustre: server umount lustre-OST0000 complete [10093.025877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10093.035081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10093.050433] Lustre: Skipped 1 previous similar message [10093.058434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10095.676518] Lustre: server umount lustre-MDT0000 complete [10098.656901] LustreError: 241334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10098.667664] LustreError: 241334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10099.434479] LustreError: 241320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785243022 with bad export cookie 3078514825577005193 [10099.439331] LustreError: 241320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10099.440498] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.688766] Lustre: server umount lustre-MDT0001 complete [10108.813921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.255259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.259911] Lustre: Skipped 2 previous similar messages [10115.115524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10119.660655] LustreError: 245517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.700320] LustreError: 245517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10125.191650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.550486] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10139.772619] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10147.175247] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10156.443396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.228959] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10171.890239] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10318.820807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.830325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10318.832978] Lustre: Skipped 1 previous similar message [10318.833753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10318.862851] Lustre: Skipped 2 previous similar messages [10324.830697] Lustre: server umount lustre-OST0000 complete [10339.297650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10339.304775] Lustre: Skipped 7 previous similar messages [10343.391172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10343.552948] Lustre: server umount lustre-MDT0000 complete [10344.423638] LustreError: 245516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.450935] LustreError: 245516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10347.344701] LustreError: 245498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785243270 with bad export cookie 3078514825577006075 [10347.347962] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10347.349623] LustreError: 245498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10347.656485] Lustre: server umount lustre-MDT0001 complete [10443.318428] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10445.774161] Key type lgssc unregistered [10446.025707] LNet: 249125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10446.044263] LNetError: 249125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10446.086948] LNet: Removed LNI 192.168.202.153@tcp [10447.262665] Key type .llcrypt unregistered [10447.265164] Key type ._llcrypt unregistered [10474.668819] Key type ._llcrypt registered [10474.672345] Key type .llcrypt registered [10474.781992] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10492.346917] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10493.664156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10493.771064] alg: No test for adler32 (adler32-zlib) [10494.907655] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [10495.274161] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10497.048716] Key type lgssc registered [10498.296668] Lustre: Echo OBD driver; http://www.lustre.org/ [10552.718820] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10564.008405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10564.078971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.354914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10565.410645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10565.551365] Lustre: lustre-MDT0000: new disk, initializing [10565.722738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10565.739852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10571.351892] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10582.981982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.129377] Lustre: 253216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10583.245826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10583.251994] Lustre: Skipped 1 previous similar message [10583.398566] Lustre: lustre-MDT0001: new disk, initializing [10583.586265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10583.653405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10583.680337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10588.674160] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10594.057866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10601.368566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.831419] Lustre: lustre-OST0000: new disk, initializing [10601.839525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10601.850220] Lustre: 254849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10602.003759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10606.663902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10606.672263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10606.867343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10609.618623] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10621.030029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.261458] Lustre: lustre-OST0001: new disk, initializing [10621.265603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10621.284274] Lustre: 255719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10621.429729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10628.792023] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10629.233041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10629.245802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10629.329858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10642.612341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10653.015554] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10670.563535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10670.568599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.571541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10670.594591] Lustre: Skipped 2 previous similar messages [10672.207553] Lustre: server umount lustre-MDT0000 complete [10676.462403] LustreError: 253209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785243599 with bad export cookie 8924263101994331085 [10676.469720] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10676.480840] LustreError: 253209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10676.808556] Lustre: server umount lustre-MDT0001 complete [10691.865700] Lustre: server umount lustre-OST0000 complete [10705.344645] Lustre: server umount lustre-OST0001 complete [10719.551578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10730.125605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10738.369650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10749.552521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10769.782514] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10783.748990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.761808] Lustre: lustre-MDT0000: reset Object Index mappings [10784.044401] LustreError: 260739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.056755] LustreError: 260739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10784.119765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10789.344238] LustreError: 260740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.578451] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10794.467125] LustreError: 260739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.588170] LustreError: 260740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.034432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.051067] Lustre: lustre-MDT0001: reset Object Index mappings [10801.465914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.741130] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10808.947283] Lustre: 261882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10816.410626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.264224] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10826.856843] LustreError: 262237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10829.948860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10833.651677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.923705] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10833.930859] Lustre: Skipped 1 previous similar message [10839.045669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10840.982726] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10849.646630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10854.102675] Lustre: 263756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10860.112265] Lustre: server umount lustre-MDT0000 complete [10863.076675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10863.090754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.112287] Lustre: Skipped 1 previous similar message [10863.117911] LustreError: 262003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.137928] LustreError: 262003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10864.628164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.657176] Lustre: Skipped 2 previous similar messages [10864.841886] LustreError: 263774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785243787 with bad export cookie 8924263101994337875 [10864.848913] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10864.851388] LustreError: 263774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10865.221976] Lustre: server umount lustre-MDT0001 complete [10880.140849] Lustre: server umount lustre-OST0000 complete [10894.417972] Lustre: server umount lustre-OST0001 complete [10915.694174] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:03:56 (1785243836) [10934.134523] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10944.938983] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10985.414788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10985.625279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10985.643188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10985.687235] Lustre: lustre-MDT0000: new disk, initializing [10985.751432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10985.769695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10990.576685] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11002.809700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.888382] Lustre: 268020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11002.929538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11002.931469] Lustre: Skipped 1 previous similar message [11002.973438] Lustre: lustre-MDT0001: new disk, initializing [11003.089790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11003.102486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11008.253766] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11017.278693] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11022.488847] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11029.349027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11029.776212] Lustre: lustre-OST0000: new disk, initializing [11029.782726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11029.794553] Lustre: 269987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11031.932356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11031.942967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11032.117044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11036.840968] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11047.097568] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11052.405617] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11052.731771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11057.899951] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11058.103694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11062.752681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11062.753238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11062.762183] LustreError: Skipped 1 previous similar message [11062.765208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11062.765217] Lustre: Skipped 3 previous similar messages [11062.810813] Lustre: Skipped 1 previous similar message [11066.624692] Lustre: server umount lustre-OST0000 complete [11072.997666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11073.017739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11073.027832] Lustre: Skipped 1 previous similar message [11073.048216] Lustre: Skipped 2 previous similar messages [11076.798192] Lustre: server umount lustre-MDT0000 complete [11078.112218] LustreError: 268027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11078.144475] LustreError: 268027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11081.681081] LustreError: 268013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785244004 with bad export cookie 8924263101994339968 [11081.686771] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.696643] LustreError: 268013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11082.131615] Lustre: server umount lustre-MDT0001 complete [11103.048355] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11115.278742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.808829] LustreError: 272919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.872830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11115.876426] Lustre: Skipped 2 previous similar messages [11122.457516] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11133.393453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11137.750316] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11141.210754] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11150.719032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11159.724717] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11170.266422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11170.654670] Lustre: lustre-OST0001: new disk, initializing [11170.665596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11170.677505] Lustre: 275231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11172.290374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11172.297209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11172.406330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11177.870972] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11189.343866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11193.952452] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11199.752578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11199.998773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11208.178263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.189871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.195577] Lustre: Skipped 3 previous similar messages [11208.216628] Lustre: Skipped 3 previous similar messages [11213.285412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11213.642760] Lustre: server umount lustre-MDT0000 complete [11217.780810] LustreError: 272900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785244140 with bad export cookie 8924263101994340850 [11217.793607] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11218.059627] Lustre: server umount lustre-MDT0001 complete [11233.088369] Lustre: server umount lustre-OST0000 complete [11234.787718] Lustre: 250706:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785244141/real 1785244141] req@ffff899a52029c00 x1871963402518016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785244157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11234.812859] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.601274] Lustre: server umount lustre-OST0001 complete [11258.918795] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11271.677124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.157049] LustreError: 278289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.181566] LustreError: 278289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11272.332870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11272.341521] Lustre: Skipped 3 previous similar messages [11277.356311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11287.872189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.037695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11288.046825] Lustre: Skipped 3 previous similar messages [11294.233421] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11298.112929] Lustre: 279431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11305.937515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11314.251134] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11317.788743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11324.427268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11330.089830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11331.626143] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11340.591955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11345.061664] Lustre: 281307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11350.902539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11350.960437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11361.138604] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:11:22 (1785244282) [11376.104470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.107192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11376.124222] Lustre: Skipped 2 previous similar messages [11376.134940] Lustre: Skipped 3 previous similar messages [11376.408755] Lustre: server umount lustre-MDT0000 complete [11380.635664] LustreError: 278270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785244303 with bad export cookie 8924263101994347402 [11380.643279] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11380.670818] LustreError: 278270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11383.181843] Lustre: server umount lustre-MDT0001 complete [11393.525315] Lustre: server umount lustre-OST0000 complete [11397.702597] Lustre: server umount lustre-OST0001 complete [11404.816838] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [11415.961677] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11471.924809] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11484.623284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.781797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11484.797640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11484.844268] Lustre: lustre-MDT0000: new disk, initializing [11484.909944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11489.782311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11501.009390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11501.215384] Lustre: 285547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11501.607348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11506.539422] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11519.025869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.292402] Lustre: 287181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11521.340777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11526.220740] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11537.183136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11537.373697] Lustre: 288051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11537.481326] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11537.485486] Lustre: Skipped 6 previous similar messages [11539.350292] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11539.366052] Lustre: Skipped 1 previous similar message [11539.550697] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11544.435463] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11553.976694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11557.814577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11557.817318] Lustre: Skipped 1 previous similar message [11574.896660] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:14:55 (1785244495) [11578.340200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11578.346271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.356530] Lustre: Skipped 3 previous similar messages [11578.360297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11578.364688] Lustre: Skipped 2 previous similar messages [11579.323723] Lustre: server umount lustre-MDT0000 complete [11580.402224] LustreError: 285558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.427371] LustreError: 285558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11583.321907] LustreError: 285538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785244506 with bad export cookie 8924263101994355053 [11583.347281] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11583.656176] Lustre: server umount lustre-MDT0001 complete [11598.544428] Lustre: server umount lustre-OST0000 complete [11601.696162] Lustre: 250705:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785244508/real 1785244508] req@ffff8999432ff480 x1871963402669312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785244524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11604.048843] Lustre: server umount lustre-OST0001 complete [11615.940881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.762467] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11633.158386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.438096] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11649.748779] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11657.208240] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11665.166373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11673.011533] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11681.001555] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11703.859582] loop1: detected capacity change from 0 to 204800000 [11704.080790] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11704.087822] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11704.094566] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11709.908370] loop1: detected capacity change from 0 to 204800000 [11715.879619] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.988979] LustreError: 290290:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13675: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11715.997643] LustreError: 290290:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13675 [11716.001162] LustreError: 290290:0:(mgs_handler.c:517:mgs_target_reg()) fs13675-MDT0001: Error checking target index: rc = -13 [11716.010976] LustreError: fs13675-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11716.018243] LustreError: 294022:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11716.022593] LustreError: 294022:0:(tgt_mount.c:2129:server_put_super()) no obd fs13675-MDT0001 [11716.033432] LustreError: 294022:0:(tgt_mount.c:116:server_deregister_mount()) fs13675-MDT0001 not registered [11716.069207] Lustre: server umount fs13675-MDT0001 complete [11716.072144] LustreError: 294022:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11725.691506] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.956127] Lustre: Setting parameter fs13675-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13675-MDT0001 [11726.000610] Lustre: srv-fs13675-MDT0001: No data found on store. Initialize space. [11726.004411] Lustre: Skipped 4 previous similar messages [11726.100512] Lustre: fs13675-MDT0001: new disk, initializing [11726.103666] Lustre: Skipped 3 previous similar messages [11726.233828] Lustre: srv-fs13675-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11726.244094] Lustre: 294489:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13675-MDT0001-mdtlov: can't create new llog: rc = -115 [11726.256421] LustreError: 294489:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13675-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11731.676553] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11739.643736] Lustre: server umount fs13675-MDT0001 complete [11755.013344] loop1: detected capacity change from 0 to 204800000 [11758.064707] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11760.820794] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11761.216640] Lustre: 296059:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13675-MDT0001-mdtlov: can't create new llog: rc = -115 [11761.236852] LustreError: 296059:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13675-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11766.140524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11771.920636] Lustre: server umount fs13675-MDT0001 complete [11783.821950] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [11787.233226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11787.237492] LustreError: Skipped 1 previous similar message [11787.239508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.255322] Lustre: Skipped 4 previous similar messages [11790.112912] Lustre: server umount lustre-MDT0000 complete [11799.130140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.308431] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11803.922236] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11804.659921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11812.934800] loop1: detected capacity change from 0 to 204800000 [11819.345936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11819.746891] Lustre: 298507:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13675-MDT0001-mdtlov: can't create new llog: rc = -115 [11819.759205] LustreError: 298507:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13675-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11826.245676] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11832.751119] Lustre: server umount fs13675-MDT0001 complete [11840.138595] LustreError: 292131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.158776] LustreError: 292131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 102 previous similar messages [11858.405747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11871.202053] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11871.550623] Lustre: server umount lustre-OST0000 complete [11881.744081] Lustre: server umount lustre-MDT0000 complete [11885.443177] LustreError: 290284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785244808 with bad export cookie 8924263101994357734 [11885.457124] LustreError: 290284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11885.466593] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11885.816732] Lustre: server umount lustre-MDT0001 complete [11899.273344] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11902.608661] Key type lgssc unregistered [11902.978634] LNet: 300854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11902.991900] LNetError: 300854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11904.038707] LNet: Removed LNI 192.168.202.153@tcp [11905.069157] Key type .llcrypt unregistered [11905.072618] Key type ._llcrypt unregistered [11924.369912] Key type ._llcrypt registered [11924.371634] Key type .llcrypt registered [11924.483771] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11925.938740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11926.040847] alg: No test for adler32 (adler32-zlib) [11927.079703] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11927.279173] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11928.943706] Key type lgssc registered [11929.723421] Lustre: Echo OBD driver; http://www.lustre.org/ [11936.393469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11936.428862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.941772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.382760] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11947.506125] LustreError: 302127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.525866] LustreError: 302127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11951.358927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.817596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11956.691177] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11963.919352] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11969.836632] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11975.958867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11976.350411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11977.386876] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.518822] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.554783] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11982.724342] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11987.944021] LustreError: 303955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.979531] LustreError: 303955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11990.848736] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11993.077379] LustreError: 304037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.092243] LustreError: 304037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12003.315134] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.347483] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12009.746216] loop1: detected capacity change from 0 to 204800000 [12009.958312] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12009.968359] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12009.987451] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12016.362536] loop1: detected capacity change from 0 to 204800000 [12019.332856] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.367936] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [12022.535571] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12022.774715] Lustre: fs1430-OST0000: new disk, initializing [12022.788259] Lustre: srv-fs1430-OST0000: No data found on store. Initialize space. [12022.805204] Lustre: 305861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12022.920032] Lustre: fs1430-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12030.052677] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12038.941095] Lustre: server umount fs1430-OST0000 complete [12052.266168] loop1: detected capacity change from 0 to 204800000 [12054.524669] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.541920] LustreError: 303956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [12058.364907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12058.538169] Lustre: fs1430-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12064.746423] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12065.935906] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12075.345274] loop2: detected capacity change from 0 to 204800000 [12075.698269] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12075.710250] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12075.722673] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12081.517698] loop2: detected capacity change from 0 to 204800000 [12087.563064] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12087.664240] LustreError: fs1430-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12087.684686] LustreError: fs1430-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12087.694507] LustreError: 308902:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12087.703184] LustreError: 308902:0:(tgt_mount.c:2129:server_put_super()) no obd fs1430-OST0001 [12087.717221] LustreError: 308902:0:(tgt_mount.c:116:server_deregister_mount()) fs1430-OST0001 not registered [12087.780293] Lustre: server umount fs1430-OST0001 complete [12087.786109] LustreError: 308902:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12100.366763] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.612759] Lustre: fs1430-OST0001: new disk, initializing [12100.615978] Lustre: srv-fs1430-OST0001: No data found on store. Initialize space. [12100.623177] Lustre: 309454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12100.694624] Lustre: fs1430-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12107.939690] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12116.669505] Lustre: server umount fs1430-OST0000 complete [12121.071311] LustreError: 303957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12121.103484] LustreError: 303957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12126.361054] Lustre: server umount fs1430-OST0001 complete [12140.048613] loop1: detected capacity change from 0 to 204800000 [12146.444160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12146.685033] Lustre: fs1430-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12153.656901] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12161.415641] loop2: detected capacity change from 0 to 204800000 [12167.532504] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12175.212150] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12187.347706] Lustre: server umount fs1430-OST0000 complete [12198.930649] Lustre: server umount fs1430-OST0001 complete [12213.223058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.224597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12213.264575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12213.280871] LustreError: Skipped 1 previous similar message [12218.079040] Lustre: server umount lustre-OST0000 complete [12222.952594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12222.966127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12222.993849] Lustre: Skipped 1 previous similar message [12223.003149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12223.013057] Lustre: Skipped 1 previous similar message [12228.578167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12228.592570] Lustre: Skipped 3 previous similar messages [12233.697675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12233.705089] Lustre: Skipped 1 previous similar message [12236.255898] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12236.549909] Lustre: server umount lustre-MDT0000 complete [12241.248751] LustreError: 302107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785245164 with bad export cookie 8516121808661917484 [12241.252526] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12241.259165] LustreError: 302107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12241.778322] Lustre: server umount lustre-MDT0001 complete [12256.463533] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12260.375167] Key type lgssc unregistered [12260.774635] LNet: 315383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12260.802928] LNetError: 315383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12261.863751] LNet: Removed LNI 192.168.202.153@tcp [12263.360426] Key type .llcrypt unregistered [12263.362779] Key type ._llcrypt unregistered [12306.055279] Key type ._llcrypt registered [12306.058210] Key type .llcrypt registered [12306.223597] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [12323.779054] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12324.907742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12324.996231] alg: No test for adler32 (adler32-zlib) [12326.292072] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [12326.668658] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12328.367162] Key type lgssc registered [12329.653790] Lustre: Echo OBD driver; http://www.lustre.org/ [12372.293699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12372.325652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12373.621575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12373.673948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12373.789809] Lustre: lustre-MDT0000: new disk, initializing [12373.876377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12373.905152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12378.798130] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12389.998813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.162571] Lustre: 319245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12390.277909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12390.283165] Lustre: Skipped 1 previous similar message [12390.389872] Lustre: lustre-MDT0001: new disk, initializing [12390.529651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12390.567050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12390.577946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12395.427175] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12407.120711] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12414.213280] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12422.328606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12422.624359] Lustre: lustre-OST0000: new disk, initializing [12422.629267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12422.636621] Lustre: 321214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12422.747619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.396562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12428.410761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12428.585132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12430.040788] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12441.204899] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12443.639135] Lustre: Mounted lustre-client [12444.783232] Lustre: Unmounted lustre-client [12454.370439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12454.370556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.379339] LustreError: Skipped 1 previous similar message [12454.380569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12454.386930] Lustre: Skipped 1 previous similar message [12458.984221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12458.990076] Lustre: Skipped 2 previous similar messages [12460.781982] Lustre: server umount lustre-OST0000 complete [12467.168853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12467.171316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.200742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12469.220590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12469.226435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12469.251140] Lustre: Skipped 1 previous similar message [12471.430739] Lustre: server umount lustre-MDT0000 complete [12474.336547] LustreError: 319255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.375539] LustreError: 319255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12476.789195] LustreError: 321212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785245399 with bad export cookie 804706549321355257 [12476.796124] LustreError: 321212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12476.799563] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.135742] Lustre: server umount lustre-MDT0001 complete [12490.211721] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12492.497962] Key type lgssc unregistered [12495.905916] LNet: 323381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12495.917590] LNetError: 323381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12575.847532] LNet: Removed LNI 192.168.202.153@tcp [12576.929242] Key type .llcrypt unregistered [12576.931773] Key type ._llcrypt unregistered [12598.916821] Key type ._llcrypt registered [12598.922727] Key type .llcrypt registered [12599.051635] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [12617.422480] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12618.982021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12619.273703] alg: No test for adler32 (adler32-zlib) [12620.462711] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [12620.755240] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12622.525071] Key type lgssc registered [12624.264609] Lustre: Echo OBD driver; http://www.lustre.org/ [12663.804127] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:33:04 (1785245584) [12671.384296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12671.424498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.702827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12672.743360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12672.822147] Lustre: lustre-MDT0000: new disk, initializing [12672.964339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12673.006204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12678.676924] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12691.161143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12691.338165] Lustre: 326610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12691.395892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12691.400221] Lustre: Skipped 1 previous similar message [12691.503286] Lustre: lustre-MDT0001: new disk, initializing [12691.597676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12691.653170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12691.666884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12697.702393] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12710.264627] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12717.909946] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12727.615383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12728.072965] Lustre: lustre-OST0000: new disk, initializing [12728.076566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12728.087408] Lustre: 328579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12728.189601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12734.574757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12734.600452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12734.755179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12736.439557] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12748.560223] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12765.671661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.683868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12765.694859] Lustre: Skipped 1 previous similar message [12765.695426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12765.719408] Lustre: Skipped 1 previous similar message [12770.798099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12770.826985] Lustre: Skipped 1 previous similar message [12771.468754] Lustre: server umount lustre-OST0000 complete [12778.988189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12778.999231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.020196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.025782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12781.029828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.040405] Lustre: Skipped 1 previous similar message [12783.162146] Lustre: server umount lustre-MDT0000 complete [12787.541780] LustreError: 326600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785245710 with bad export cookie 14545433963260956470 [12787.558627] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12787.962216] Lustre: server umount lustre-MDT0001 complete [12800.716269] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12804.347606] Key type lgssc unregistered [12804.718728] LNet: 330486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12804.724627] LNetError: 330486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12804.745356] LNet: Removed LNI 192.168.202.153@tcp [12805.839387] Key type .llcrypt unregistered [12805.841310] Key type ._llcrypt unregistered [12830.353566] Key type ._llcrypt registered [12830.357501] Key type .llcrypt registered [12830.459663] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12831.615237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12831.659511] alg: No test for adler32 (adler32-zlib) [12832.648494] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [12832.831713] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12834.447602] Key type lgssc registered [12835.675589] Lustre: Echo OBD driver; http://www.lustre.org/ [12847.912608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12847.951994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12849.846199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12858.132588] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12859.367372] LustreError: 331763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12859.397790] LustreError: 331763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12864.494941] LustreError: 331764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.606706] LustreError: 331763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.053372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12870.764441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12877.869541] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12888.948906] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12895.845519] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12904.577792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12905.160410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12914.589979] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12924.850621] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12938.052876] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12939.971081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12945.376602] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12945.388979] Lustre: Skipped 1 previous similar message [12945.393246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12945.424651] Lustre: Skipped 1 previous similar message [12948.619094] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [12950.515492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12950.529790] Lustre: Skipped 2 previous similar messages [12953.717087] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [12958.837697] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [12958.845970] Lustre: Skipped 3 previous similar messages [12959.199116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12960.738280] LustreError: 331763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12960.762542] LustreError: 331763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12961.395886] Lustre: server umount lustre-MDT0000 complete [12965.860064] LustreError: 333071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.872123] LustreError: 333071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12966.444245] LustreError: 331744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785245889 with bad export cookie 6704521718528562391 [12966.459234] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12966.475349] LustreError: 331744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12969.092973] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [12970.978957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12973.092581] Lustre: server umount lustre-MDT0001 complete [12978.244996] Lustre: server umount lustre-OST0000 complete [12988.239747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12992.621692] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12995.316645] Lustre: 335952:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.153@tcp [12997.058803] Lustre: server umount MGS complete [13007.017970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.658232] LustreError: 336525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.675944] LustreError: 336525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13007.751829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13013.444099] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13024.249494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13024.650613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13030.091178] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13040.428696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13040.702703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13048.526295] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13060.899324] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13064.145493] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13072.297502] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13074.718755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13080.544092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13080.552109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13080.561266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13080.567167] Lustre: Skipped 2 previous similar messages [13081.067668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13082.753773] Lustre: server umount lustre-OST0000 complete [13088.738159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13088.758491] Lustre: Skipped 2 previous similar messages [13094.891576] Lustre: server umount lustre-MDT0000 complete [13098.980835] LustreError: 336525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.991646] LustreError: 336525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13100.095949] LustreError: 336506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785246023 with bad export cookie 6704521718528563224 [13100.116735] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13100.120504] LustreError: 336506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13100.745451] Lustre: server umount lustre-MDT0001 complete [13114.694052] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13118.791316] Key type lgssc unregistered [13119.205903] LNet: 339964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13119.214452] LNetError: 339964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13119.245590] LNet: Removed LNI 192.168.202.153@tcp [13120.908607] Key type .llcrypt unregistered [13120.910491] Key type ._llcrypt unregistered [13142.773447] Key type ._llcrypt registered [13142.777715] Key type .llcrypt registered [13142.914785] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [13161.200952] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [13163.020115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13163.085135] alg: No test for adler32 (adler32-zlib) [13164.305580] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [13164.652618] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [13166.367417] Key type lgssc registered [13167.888764] Lustre: Echo OBD driver; http://www.lustre.org/ [13211.374942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13211.400638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.816568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13212.859938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13212.960689] Lustre: lustre-MDT0000: new disk, initializing [13213.099580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13213.131927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13220.410141] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13235.441404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13235.546684] Lustre: 343126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13235.611654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13235.614425] Lustre: Skipped 1 previous similar message [13235.692629] Lustre: lustre-MDT0001: new disk, initializing [13235.831718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13235.893858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13235.908794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13242.416173] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13253.985454] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13261.138035] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13269.727349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13270.252739] Lustre: lustre-OST0000: new disk, initializing [13270.264700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13270.269146] Lustre: 345099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13270.370695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13278.419254] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13278.783925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13278.793843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13278.976447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13289.080802] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13295.563185] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13295.952868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13302.487152] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13302.954726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13309.921251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13309.933613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13309.939242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13309.979735] LustreError: Skipped 1 previous similar message [13312.334973] Lustre: server umount lustre-OST0000 complete [13317.607274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13317.625634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13317.649842] Lustre: Skipped 1 previous similar message [13317.661042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13317.670578] Lustre: Skipped 1 previous similar message [13320.165080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13320.166229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13323.783138] Lustre: server umount lustre-MDT0000 complete [13329.027239] LustreError: 343121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785246252 with bad export cookie 12582115309085982044 [13329.038669] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13329.053390] LustreError: 343121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13329.454855] Lustre: server umount lustre-MDT0001 complete [13342.409528] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:44:22 (1785246262) [13363.219172] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [13377.028745] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [13428.348406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13428.670769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13428.692684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13428.766880] Lustre: lustre-MDT0000: new disk, initializing [13428.862344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13428.890940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13435.024145] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13448.348421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13448.515340] Lustre: 349865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13448.605365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13448.609769] Lustre: Skipped 1 previous similar message [13448.736458] Lustre: lustre-MDT0001: new disk, initializing [13448.898874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13448.941289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13448.949425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13453.689161] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13464.214935] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13471.823155] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13480.338899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13480.729100] Lustre: lustre-OST0000: new disk, initializing [13480.739123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13480.752808] Lustre: 351835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13480.878595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13487.198369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13487.205466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13487.370701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13488.510471] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13499.226869] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13533.673838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13533.684405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13533.686468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13533.691245] Lustre: Skipped 1 previous similar message [13533.698880] Lustre: Skipped 1 previous similar message [13537.355731] Lustre: server umount lustre-OST0000 complete [13543.905822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.909152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13543.930420] Lustre: Skipped 1 previous similar message [13543.947019] Lustre: Skipped 2 previous similar messages [13548.386320] Lustre: server umount lustre-MDT0000 complete [13549.026291] LustreError: 349876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.049438] LustreError: 349876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13553.120928] LustreError: 349870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.138700] LustreError: 349870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13553.750172] LustreError: 349857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785246476 with bad export cookie 12582115309085982926 [13553.751461] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13553.768180] LustreError: 349857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13554.202782] Lustre: server umount lustre-MDT0001 complete [13571.579701] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13575.324508] Key type lgssc unregistered [13575.704196] LNet: 354001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13575.717741] LNetError: 354001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13575.753811] LNet: Removed LNI 192.168.202.153@tcp [13577.347660] Key type .llcrypt unregistered [13577.352953] Key type ._llcrypt unregistered [13686.764210] Key type ._llcrypt registered [13686.766383] Key type .llcrypt registered [13686.879876] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13687.202291] Key type .llcrypt unregistered [13687.204175] Key type ._llcrypt unregistered [13772.297855] Key type ._llcrypt registered [13772.301223] Key type .llcrypt registered [13772.379414] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13772.593317] Key type .llcrypt unregistered [13772.594855] Key type ._llcrypt unregistered [13804.870348] Key type ._llcrypt registered [13804.872838] Key type .llcrypt registered [13805.057539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13805.614329] Key type .llcrypt unregistered [13805.617269] Key type ._llcrypt unregistered [13839.407346] Key type ._llcrypt registered [13839.410203] Key type .llcrypt registered [13839.510889] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13839.788461] Key type .llcrypt unregistered [13839.791324] Key type ._llcrypt unregistered [13928.240408] Key type ._llcrypt registered [13928.248294] Key type .llcrypt registered [13928.451131] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13928.858372] Key type .llcrypt unregistered [13928.861980] Key type ._llcrypt unregistered [13961.206837] Key type ._llcrypt registered [13961.209346] Key type .llcrypt registered [13961.336124] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13961.639265] Key type .llcrypt unregistered [13961.640346] Key type ._llcrypt unregistered [14004.532415] Key type ._llcrypt registered [14004.533475] Key type .llcrypt registered [14004.598849] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14004.881406] Key type .llcrypt unregistered [14004.886251] Key type ._llcrypt unregistered [14044.918268] Key type ._llcrypt registered [14044.922413] Key type .llcrypt registered [14045.022305] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14045.263290] Key type .llcrypt unregistered [14045.264495] Key type ._llcrypt unregistered [14081.867414] Key type ._llcrypt registered [14081.869603] Key type .llcrypt registered [14082.000624] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14082.318456] Key type .llcrypt unregistered [14082.320042] Key type ._llcrypt unregistered [14117.518364] Key type ._llcrypt registered [14117.525119] Key type .llcrypt registered [14117.616237] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14117.920305] Key type .llcrypt unregistered [14117.929495] Key type ._llcrypt unregistered [14196.272367] Key type ._llcrypt registered [14196.276631] Key type .llcrypt registered [14196.396926] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14196.807479] Key type .llcrypt unregistered [14196.809971] Key type ._llcrypt unregistered [14232.085300] Key type ._llcrypt registered [14232.089978] Key type .llcrypt registered [14232.241653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14232.580627] Key type .llcrypt unregistered [14232.589371] Key type ._llcrypt unregistered [14275.381541] Key type ._llcrypt registered [14275.384461] Key type .llcrypt registered [14275.521891] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14275.856423] Key type .llcrypt unregistered [14275.858738] Key type ._llcrypt unregistered [14338.491915] Key type ._llcrypt registered [14338.493502] Key type .llcrypt registered [14338.690358] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [14357.291428] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [14359.136950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14359.173069] alg: No test for adler32 (adler32-zlib) [14360.277109] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [14360.572579] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [14362.247232] Key type lgssc registered [14363.354784] Lustre: Echo OBD driver; http://www.lustre.org/ [14408.912962] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14408.959319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14410.339970] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14410.366746] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14410.480701] Lustre: tst250fs-MDT0000: new disk, initializing [14410.562063] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14410.573648] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14416.096089] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [14431.437437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14431.661193] Lustre: 374382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14431.748567] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14431.756643] Lustre: Skipped 1 previous similar message [14431.864491] Lustre: tst250fs-MDT0001: new disk, initializing [14431.949490] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14431.992164] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14432.004885] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14437.465345] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [14443.359648] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14451.549314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14452.220374] Lustre: tst250fs-OST0000: new disk, initializing [14452.230978] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14452.242640] Lustre: 376015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14452.398118] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14453.440193] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14453.458937] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14453.680963] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14459.992781] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [14472.338039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14472.540938] Lustre: tst250fs-OST0001: new disk, initializing [14472.544730] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14472.551924] Lustre: 376886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14472.695756] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14480.173449] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [14481.018463] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14481.035145] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14481.193536] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14487.607773] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14489.817956] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14489.825878] Lustre: Skipped 1 previous similar message [14512.108411] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14512.116575] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14512.121018] Lustre: Skipped 3 previous similar messages [14512.156834] Lustre: Skipped 3 previous similar messages [14517.218603] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14517.229808] Lustre: Skipped 2 previous similar messages [14518.048334] Lustre: server umount tst250fs-MDT0000 complete [14522.546947] LustreError: 374372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785247445 with bad export cookie 12426847346445067743 [14522.562631] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14522.569295] LustreError: 374372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14523.203840] Lustre: server umount tst250fs-MDT0001 complete [14539.000627] Lustre: server umount tst250fs-OST0000 complete [14553.773646] Lustre: server umount tst250fs-OST0001 complete [14576.750918] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [14586.374837] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [14629.388365] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:05:49 (1785247549) [14631.674362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14651.491244] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [14663.381410] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [14701.504701] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14361 sec ========= 10:07:01 (1785247621) [14703.591693] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:07:04 (1785247624) === [14707.761724] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:07:08 (1785247628) === [14754.812992] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [14757.840166] Key type lgssc unregistered [14758.269779] LNet: 386036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14758.281278] LNetError: 386036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14758.299789] LNet: Removed LNI 192.168.202.153@tcp [14759.686139] Key type .llcrypt unregistered [14759.690906] Key type ._llcrypt unregistered