[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495893656 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002453] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011420] LSM: Security Framework initializing [ 0.012097] Yama: becoming mindful. [ 0.014008] SELinux: Initializing. [ 0.015126] *** VALIDATE selinux *** [ 0.026684] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031539] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035579] *** VALIDATE tmpfs *** [ 0.037376] *** VALIDATE proc *** [ 0.038378] *** VALIDATE cgroup *** [ 0.040014] *** VALIDATE cgroup2 *** [ 0.042129] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043193] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046055] Spectre V2 : User space: Vulnerable [ 0.047015] Speculative Store Bypass: Vulnerable [ 0.051324] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.054304] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056018] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057036] ... version: 2 [ 0.058021] ... bit width: 48 [ 0.059020] ... generic registers: 4 [ 0.060022] ... value mask: 0000ffffffffffff [ 0.061024] ... max period: 00007fffffffffff [ 0.062025] ... fixed-purpose events: 3 [ 0.063019] ... event mask: 000000070000000f [ 0.064598] rcu: Hierarchical SRCU implementation. [ 0.067165] smp: Bringing up secondary CPUs ... [ 0.068941] x86: Booting SMP configuration: [ 0.069117] .... node #0, CPUs: #1 #2 #3 [ 0.087224] smp: Brought up 1 node, 4 CPUs [ 0.089021] smpboot: Max logical packages: 1 [ 0.090022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.216305] node 0 deferred pages initialised in 124ms [ 0.219008] devtmpfs: initialized [ 0.220300] x86/mm: Memory block size: 128MB [ 0.223504] gcov: version magic: 0x41383552 [ 0.225369] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.226105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.227338] pinctrl core: initialized pinctrl subsystem [ 0.228268] [ 0.229013] ************************************************************* [ 0.230017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231019] ** ** [ 0.232019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233020] ** ** [ 0.234019] ** This means that this kernel is built to expose internal ** [ 0.235305] ** IOMMU data structures, which may compromise security on ** [ 0.236023] ** your system. ** [ 0.237022] ** ** [ 0.238023] ** If you see this message and you are not debugging the ** [ 0.239026] ** kernel, report this immediately to your vendor! ** [ 0.240020] ** ** [ 0.241022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242019] ************************************************************* [ 0.245118] NET: Registered protocol family 16 [ 0.246533] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.248088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.251051] cpuidle: using governor menu [ 0.253000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255610] PCI: Using configuration type 1 for base access [ 0.264146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.280025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301090] cryptd: max_cpu_qlen set to 1000 [ 0.313356] ACPI: Added _OSI(Module Device) [ 0.314025] ACPI: Added _OSI(Processor Device) [ 0.315031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316091] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324034] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.330695] ACPI: Interpreter enabled [ 0.331886] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332020] ACPI: Using IOAPIC for interrupt routing [ 0.333121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.334386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350018] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356024] acpiphp: Slot [2] registered [ 0.357363] acpiphp: Slot [5] registered [ 0.359161] acpiphp: Slot [6] registered [ 0.360711] acpiphp: Slot [7] registered [ 0.361355] acpiphp: Slot [8] registered [ 0.362439] acpiphp: Slot [9] registered [ 0.363451] acpiphp: Slot [10] registered [ 0.364501] acpiphp: Slot [3] registered [ 0.365159] acpiphp: Slot [4] registered [ 0.366963] acpiphp: Slot [11] registered [ 0.367166] acpiphp: Slot [12] registered [ 0.368126] acpiphp: Slot [13] registered [ 0.369147] acpiphp: Slot [14] registered [ 0.370103] acpiphp: Slot [15] registered [ 0.371203] acpiphp: Slot [16] registered [ 0.372277] acpiphp: Slot [17] registered [ 0.373201] acpiphp: Slot [18] registered [ 0.374759] acpiphp: Slot [19] registered [ 0.375158] acpiphp: Slot [20] registered [ 0.376683] acpiphp: Slot [21] registered [ 0.379135] acpiphp: Slot [22] registered [ 0.383000] acpiphp: Slot [23] registered [ 0.386141] acpiphp: Slot [24] registered [ 0.399030] acpiphp: Slot [25] registered [ 0.405389] acpiphp: Slot [26] registered [ 0.411365] acpiphp: Slot [27] registered [ 0.415328] acpiphp: Slot [28] registered [ 0.422142] acpiphp: Slot [29] registered [ 0.435241] acpiphp: Slot [30] registered [ 0.439893] acpiphp: Slot [31] registered [ 0.442173] PCI host bridge to bus 0000:00 [ 0.445277] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.451095] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.456026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.463835] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.471039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.478027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.486920] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.491193] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.496470] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.518021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.535067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.545020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.548018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.555018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.562000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.575163] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.582046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.589090] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 14648 usecs [ 0.595234] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.607016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.644015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.658028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.674000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.700025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.709017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.734024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.740387] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.759021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.781019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.815032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.844000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.857018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.872023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.897019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.913649] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.924016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.933022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.967020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.977777] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.986024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.000496] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.017023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.037000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.051022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.067023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.097023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.123536] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.130492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.135768] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.136490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.139591] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.147233] iommu: Default domain type: Passthrough [ 1.148000] SCSI subsystem initialized [ 1.148000] ACPI: bus type USB registered [ 1.150169] usbcore: registered new interface driver usbfs [ 1.155289] usbcore: registered new interface driver hub [ 1.163130] usbcore: registered new device driver usb [ 1.166522] pps_core: LinuxPPS API ver. 1 registered [ 1.169016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.174148] PTP clock support registered [ 1.178056] EDAC MC: Ver: 3.0.0 [ 1.182536] PCI: Using ACPI for IRQ routing [ 1.187661] NetLabel: Initializing [ 1.191019] NetLabel: domain hash size = 128 [ 1.194018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.196191] NetLabel: unlabeled traffic allowed by default [ 1.202563] vgaarb: loaded [ 1.207600] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.210272] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.222486] clocksource: Switched to clocksource kvm-clock [ 1.425505] VFS: Disk quotas dquot_6.6.0 [ 1.429365] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.441228] *** VALIDATE ramfs *** [ 1.442710] *** VALIDATE hugetlbfs *** [ 1.448811] pnp: PnP ACPI init [ 1.461099] pnp: PnP ACPI: found 6 devices [ 1.499425] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.509223] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.516099] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.525197] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.533565] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.539220] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.544385] NET: Registered protocol family 2 [ 1.548576] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.559221] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.565322] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.573365] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.577754] TCP: Hash tables configured (established 65536 bind 65536) [ 1.583654] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.592281] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.601929] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.606518] NET: Registered protocol family 1 [ 1.610212] RPC: Registered named UNIX socket transport module. [ 1.615790] RPC: Registered udp transport module. [ 1.619223] RPC: Registered tcp transport module. [ 1.621762] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.626366] NET: Registered protocol family 44 [ 1.628259] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.634111] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.637540] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.643977] PCI: CLS 0 bytes, default 64 [ 1.653292] Unpacking initramfs... [ 4.870161] debug: unmapping init [mem 0xffff9fabbcc54000-0xffff9fabbffbffff] [ 4.883247] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.894400] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 4.906457] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.714368] Initialise system trusted keyrings [ 5.718834] Key type blacklist registered [ 5.722065] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.744449] zbud: loaded [ 5.756872] *** VALIDATE nfs *** [ 5.761062] *** VALIDATE nfs4 *** [ 5.768056] pstore: using deflate compression [ 5.784835] Platform Keyring initialized [ 6.061780] NET: Registered protocol family 38 [ 6.064761] Key type asymmetric registered [ 6.067051] Asymmetric key parser 'x509' registered [ 6.069727] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.074242] io scheduler mq-deadline registered [ 6.077079] io scheduler kyber registered [ 6.079970] io scheduler bfq registered [ 6.082318] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.086620] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.091316] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.099899] ACPI: Power Button [PWRF] [ 6.109822] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.120400] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.137751] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.161431] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.187372] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.226149] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.257129] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.262942] Non-volatile memory driver v1.3 [ 6.264574] Linux agpgart interface v0.103 [ 6.308891] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 6.313377] vda: detected capacity change from 0 to 75116544 [ 6.333876] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.337988] vdb: detected capacity change from 0 to 1073741824 [ 6.383427] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.388808] vdc: detected capacity change from 0 to 2621440000 [ 6.432030] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.435335] vdd: detected capacity change from 0 to 2621440000 [ 6.481971] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.496635] vde: detected capacity change from 0 to 4294967296 [ 6.549347] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.557039] vdf: detected capacity change from 0 to 4294967296 [ 6.581497] libphy: Fixed MDIO Bus: probed [ 6.590398] usbcore: registered new interface driver usbserial_generic [ 6.592872] usbserial: USB Serial support registered for generic [ 6.597603] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.603795] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.611154] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.614908] mousedev: PS/2 mouse device common for all mice [ 6.621497] rtc_cmos 00:05: RTC can wake from S4 [ 6.629363] rtc_cmos 00:05: registered as rtc0 [ 6.630376] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.636048] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.636090] intel_pstate: CPU model not supported [ 6.640772] hid: raw HID events driver (C) Jiri Kosina [ 6.657342] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.661468] usbcore: registered new interface driver usbhid [ 6.670587] usbhid: USB HID core driver [ 6.672411] drop_monitor: Initializing network drop monitor service [ 6.675146] Initializing XFRM netlink socket [ 6.678324] NET: Registered protocol family 10 [ 6.682891] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.685566] Segment Routing with IPv6 [ 6.691041] NET: Registered protocol family 17 [ 6.693966] mpls_gso: MPLS GSO support [ 6.700270] RAS: Correctable Errors collector initialized. [ 6.702759] AVX version of gcm_enc/dec engaged. [ 6.705565] AES CTR mode by8 optimization enabled [ 6.801822] sched_clock: Marking stable (6801675516, 0)->(8504938032, -1703262516) [ 6.809556] registered taskstats version 1 [ 6.811479] Loading compiled-in X.509 certificates [ 6.813885] zswap: loaded using pool lzo/zbud [ 6.847600] Key type big_key registered [ 6.865605] Key type encrypted registered [ 6.867732] ima: No TPM chip found, activating TPM-bypass! [ 6.870752] ima: Allocated hash algorithm: sha1 [ 6.872761] ima: No architecture policies found [ 6.877755] evm: Initialising EVM extended attributes: [ 6.880919] evm: security.selinux [ 6.882829] evm: security.ima [ 6.885248] evm: security.capability [ 6.887441] evm: HMAC attrs: 0x1 [ 6.894289] rtc_cmos 00:05: setting system clock to 2026-09-19 00:01:55 UTC (1789776115) [ 6.907717] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 6.919117] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 6.928149] Write protecting the kernel read-only data: 28672k [ 6.935954] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 6.941808] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 7.008567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.021995] systemd[1]: Detected virtualization kvm. [ 7.027551] systemd[1]: Detected architecture x86-64. [ 7.039890] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.117917] systemd[1]: No hostname configured. [ 7.121910] systemd[1]: Set hostname to . [ 7.126498] random: systemd: uninitialized urandom read (16 bytes read) [ 7.132767] systemd[1]: Initializing machine ID from random generator. [ 7.233539] random: ln: uninitialized urandom read (6 bytes read) [ 7.496572] random: systemd: uninitialized urandom read (16 bytes read) [ 7.502035] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.507986] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.519472] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.143805] device-mapper: uevent: version 1.0.3 [ 9.153458] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.213725] random: fast init done [ 11.233634] virtio_net virtio0 ens2: renamed from eth0 [ 11.474956] scsi host0: ata_piix [ 11.506389] scsi host1: ata_piix [ 11.508029] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.511392] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.410076] random: crng init done [ 15.411557] random: 7 urandom warning(s) missed due to ratelimiting [ 17.801961] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.234555] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.002671] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.946933] SELinux: Disabled at runtime. [ 24.061344] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.083199] systemd[1]: Detected virtualization kvm. [ 24.096705] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.915383] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.920710] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.933631] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.944772] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.952400] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.966796] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.976633] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slic[ 26.393248] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.379221] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.197715] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.320555] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.834156] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.956308] EDAC sbridge: Ver: 1.1.2 [ 32.465157] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 33.114672] NFS: Registering the id_resolver key type [ 33.124085] Key type id_resolver registered [ 33.129269] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 87.633895] libcfs: loading out-of-tree module taints kernel. [ 87.685419] Key type ._llcrypt registered [ 87.686651] Key type .llcrypt registered [ 87.784567] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 106.990740] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 108.346245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.366386] alg: No test for adler32 (adler32-zlib) [ 109.761517] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 110.610779] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 112.320053] Key type lgssc registered [ 113.768737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.013181] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.187480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 168.749010] hrtimer: interrupt took 8407253 ns [ 181.593943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.613586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.790940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.825535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.889152] Lustre: lustre-MDT0000: new disk, initializing [ 182.970715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.995452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.258711] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.385855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.532802] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.587364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.597847] Lustre: Skipped 1 previous similar message [ 199.758157] Lustre: lustre-MDT0001: new disk, initializing [ 199.838431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.892552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.903782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.353630] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.950850] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 223.132293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.667801] Lustre: lustre-OST0000: new disk, initializing [ 223.683828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.694528] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.866209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.598098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.610387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.672303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.049706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.326149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.474204] Lustre: lustre-OST0001: new disk, initializing [ 242.477287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.482710] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.538433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.582619] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.438942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.453988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.592965] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.996872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.980909] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.070544] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.303314] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 280.930262] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 283.661466] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 286.493559] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 288.561271] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 18 20:06:34 EDT 2026 [ 307.925740] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 309.557573] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.166105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.175078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.186771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.181472] Lustre: server umount lustre-MDT0000 complete [ 335.366882] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776443 with bad export cookie 5505667161776527631 [ 335.373092] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.377050] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.912683] LustreError: 8506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.936699] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.938450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.938456] Lustre: Skipped 3 previous similar messages [ 338.942164] LustreError: 8506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 339.012306] Lustre: Skipped 4 previous similar messages [ 340.961272] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.962056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.006400] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 341.032128] Lustre: Skipped 2 previous similar messages [ 342.029038] Lustre: server umount lustre-MDT0001 complete [ 346.873978] Lustre: server umount lustre-OST0000 complete [ 350.523546] Lustre: server umount lustre-OST0001 complete [ 356.960464] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 366.383480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 401.049524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.243202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.271071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.350351] Lustre: lustre-MDT0000: new disk, initializing [ 401.434197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.450945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.079846] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 415.640683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.795281] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.906302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.923924] Lustre: Skipped 1 previous similar message [ 416.039121] Lustre: lustre-MDT0001: new disk, initializing [ 416.162502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.221199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.236398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 420.685858] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 429.274842] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.835927] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.265279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.461900] Lustre: lustre-OST0000: new disk, initializing [ 441.467824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.476625] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.587132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.505875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.514708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.657884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.713318] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 456.938658] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.889537] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.091610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.509670] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.789930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.475801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.478079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.493463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.505420] Lustre: Skipped 1 previous similar message [ 469.516604] Lustre: Skipped 1 previous similar message [ 474.602429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.614470] Lustre: Skipped 1 previous similar message [ 474.853293] Lustre: server umount lustre-OST0000 complete [ 479.714389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.844414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.850064] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.856201] Lustre: Skipped 2 previous similar messages [ 484.958445] Lustre: server umount lustre-MDT0000 complete [ 488.668587] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776597 with bad export cookie 5505667161776529164 [ 488.672483] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.684628] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 489.069818] Lustre: server umount lustre-MDT0001 complete [ 494.347981] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:10:01 (1789776601) [ 510.924116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.298724] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.356341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.150750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 516.576943] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.316762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.534865] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 534.653751] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.808683] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.110801] Lustre: Failing over lustre-MDT0000 [ 544.226458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 544.240882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.263374] Lustre: Skipped 1 previous similar message [ 544.276812] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.303622] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 545.382979] Lustre: server umount lustre-MDT0000 complete [ 561.632549] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789776654/real 1789776654] req@ffff9fab0303a680 x1876716388769920/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789776670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 561.658281] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.672102] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.684222] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 564.088319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.872184] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fab03038380 x1876716388772736/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 577.104268] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 577.508681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 577.513531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 577.548606] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 586.734634] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 590.747935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.767060] Lustre: Skipped 1 previous similar message [ 590.774278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.781751] Lustre: Skipped 1 previous similar message [ 605.151112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 605.454904] Lustre: server umount lustre-MDT0000 complete [ 609.253024] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.271696] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 609.285995] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776717 with bad export cookie 5505667161776530578 [ 609.304708] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.311473] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.819967] Lustre: server umount lustre-MDT0001 complete [ 618.963540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.507935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.519520] Lustre: Skipped 2 previous similar messages [ 623.537208] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 631.945461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.765560] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 644.466748] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 650.669982] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.115573] Lustre: Failing over lustre-MDT0000 [ 653.402827] Lustre: server umount lustre-MDT0000 complete [ 654.817106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.837139] Lustre: Skipped 2 previous similar messages [ 671.135246] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789776763/real 1789776763] req@ffff9fab03700e00 x1876716388800512/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789776779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 671.158918] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.329303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.322122] LustreError: 23249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.330545] LustreError: 23249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 680.415679] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fab03701880 x1876716388803456/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.629699] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.152678] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 686.063841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.086854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 686.099247] Lustre: Skipped 1 previous similar message [ 686.121662] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 695.995174] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 701.408788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.414275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.417090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.423254] Lustre: Skipped 1 previous similar message [ 701.429130] Lustre: Skipped 6 previous similar messages [ 714.207897] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 714.396369] Lustre: server umount lustre-MDT0000 complete [ 718.205033] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776826 with bad export cookie 5505667161776531516 [ 718.209318] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.212742] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 718.498888] Lustre: server umount lustre-MDT0001 complete [ 727.513932] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:13:54 (1789776834) [ 746.417983] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 757.122370] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 802.418707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 815.928824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.245641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 816.284467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 816.425850] Lustre: lustre-MDT0000: new disk, initializing [ 816.527424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.530719] Lustre: Skipped 2 previous similar messages [ 816.575194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 820.729099] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 832.966813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.116241] Lustre: 29633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 833.372942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 837.927897] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 849.398458] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 855.829855] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 863.340766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.583338] Lustre: 31600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 865.519356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 865.603508] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 865.632948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 865.660014] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 870.692880] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 880.164350] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 886.326725] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 906.725841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.730709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.743705] LustreError: Skipped 1 previous similar message [ 906.745853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.745862] Lustre: Skipped 7 previous similar messages [ 906.796989] Lustre: Skipped 1 previous similar message [ 912.599188] Lustre: server umount lustre-OST0000 complete [ 921.914539] Lustre: server umount lustre-MDT0000 complete [ 922.086247] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.106955] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 924.917515] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777033 with bad export cookie 5505667161776531922 [ 924.924376] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.925394] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 925.217780] Lustre: server umount lustre-MDT0001 complete [ 935.083234] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 937.160273] Key type lgssc unregistered [ 937.374595] LNet: 33578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.388292] LNetError: 33578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.397763] LNet: Removed LNI 192.168.203.142@tcp [ 937.993154] Key type .llcrypt unregistered [ 937.994808] Key type ._llcrypt unregistered [ 963.261308] Key type ._llcrypt registered [ 963.263760] Key type .llcrypt registered [ 963.383608] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 964.068932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.086623] alg: No test for adler32 (adler32-zlib) [ 965.464556] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 965.783295] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 967.439216] Key type lgssc registered [ 968.403278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.771332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.788624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.211391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.606787] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 986.596925] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.631737] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 989.329583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.576848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.814278] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 999.917552] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1005.631355] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.894031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1013.551837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.583800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1020.025889] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1021.797797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1026.868340] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.351792] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:19:06 (1789777146) [ 1040.848077] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1042.461215] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1048.290637] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:19:14 (1789777154) [ 1056.112178] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:19:22 (1789777162) [ 1056.992075] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1064.961904] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1064.971205] Lustre: Skipped 2 previous similar messages [ 1066.921856] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:19:33 (1789777173) [ 1069.005107] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1069.015903] Lustre: Skipped 2 previous similar messages [ 1086.452838] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1086.458484] Lustre: Skipped 7 previous similar messages [ 1088.875356] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:19:55 (1789777195) [ 1091.485835] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1094.703245] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1103.009134] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:20:09 (1789777209) [ 1104.816704] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1104.822256] Lustre: Skipped 3 previous similar messages [ 1116.198314] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:20:22 (1789777222) [ 1132.518198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.530290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.545227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.570225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.575871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.583383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.614650] Lustre: Skipped 1 previous similar message [ 1139.680650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.695993] Lustre: Skipped 1 previous similar message [ 1144.288063] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1144.540789] Lustre: server umount lustre-OST0000 complete [ 1149.921031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.937741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.839086] Lustre: server umount lustre-MDT0000 complete [ 1155.044620] LustreError: 35717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.056396] LustreError: 35717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1158.446378] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777267 with bad export cookie 12362983478473686352 [ 1158.456486] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1158.460804] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.928062] Lustre: server umount lustre-MDT0001 complete [ 1169.177793] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1172.060916] Key type lgssc unregistered [ 1172.329799] LNet: 42831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.349622] LNetError: 42831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.364208] LNet: Removed LNI 192.168.203.142@tcp [ 1173.206450] Key type .llcrypt unregistered [ 1173.210061] Key type ._llcrypt unregistered [ 1189.802381] Key type ._llcrypt registered [ 1189.805560] Key type .llcrypt registered [ 1189.942444] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1204.695753] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1205.492789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.565791] alg: No test for adler32 (adler32-zlib) [ 1206.555871] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 1206.772459] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1208.497200] Key type lgssc registered [ 1209.552582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.043917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.064611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.282942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1243.313162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1243.362365] Lustre: lustre-MDT0000: new disk, initializing [ 1243.464118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.474890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1247.033934] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1256.656979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.772780] Lustre: 45980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1256.829925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1256.841578] Lustre: Skipped 1 previous similar message [ 1256.918817] Lustre: lustre-MDT0001: new disk, initializing [ 1257.037527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.078730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1257.085222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1261.216994] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1270.717890] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.534473] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.906945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.128849] Lustre: lustre-OST0000: new disk, initializing [ 1284.133319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1284.138726] Lustre: 47945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1284.225038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.032825] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1294.401588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1294.417278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1294.520934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1299.501317] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1304.812113] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1305.019813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1310.062522] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1310.307635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1315.296248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.316199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.319387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.338376] Lustre: Skipped 1 previous similar message [ 1315.351797] Lustre: Skipped 1 previous similar message [ 1318.859459] Lustre: server umount lustre-OST0000 complete [ 1323.491239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.509468] LustreError: Skipped 1 previous similar message [ 1323.516437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.538849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.544773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.558663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.746375] Lustre: server umount lustre-MDT0000 complete [ 1332.578605] LustreError: 46784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777441 with bad export cookie 3676151494179101247 [ 1332.581704] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.599321] LustreError: 46784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1332.993199] Lustre: server umount lustre-MDT0001 complete [ 1342.461930] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:24:08 (1789777448) [ 1349.172148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.838906] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.864656] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1349.958406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.759293] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1355.236386] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.358842] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.507631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.855353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.495842] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1376.543873] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1382.147992] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.499320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.936942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.345179] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1403.959834] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.795535] LustreError: 52938:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1413.785837] Lustre: Modifying parameter general.timeout=1 in log params [ 1414.928911] Lustre: Modifying parameter general.timeout=2 in log params [ 1417.927550] Lustre: Modifying parameter general.timeout=5 in log params [ 1417.932839] Lustre: Skipped 2 previous similar messages [ 1422.720733] Lustre: Modifying parameter general.timeout=10 in log params [ 1422.729684] Lustre: Skipped 4 previous similar messages [ 1430.941471] Lustre: Modifying parameter general.timeout=20 in log params [ 1430.944474] Lustre: Skipped 9 previous similar messages [ 1447.230269] Lustre: Modifying parameter general.timeout=39 in log params [ 1447.236973] Lustre: Skipped 18 previous similar messages [ 1479.332556] Lustre: Modifying parameter general.timeout=76 in log params [ 1479.336662] Lustre: Skipped 36 previous similar messages [ 1537.223257] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:27:24 (1789777644) [ 1539.948645] Lustre: 59877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1541.706555] Lustre: 59989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1544.265354] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1544.277468] Lustre: Skipped 58 previous similar messages [ 1545.180539] Lustre: 60187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1545.193632] Lustre: 60187:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1575.497569] Lustre: 61736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1575.507546] Lustre: 61736:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1575.521497] LustreError: 61736:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1579.536922] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:28:06 (1789777686) [ 1592.806154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.807108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.828401] Lustre: Skipped 2 previous similar messages [ 1592.854814] Lustre: Skipped 3 previous similar messages [ 1597.921625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.929019] Lustre: Skipped 2 previous similar messages [ 1598.078225] Lustre: server umount lustre-MDT0000 complete [ 1601.592434] LustreError: 50390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777710 with bad export cookie 3676151494179102129 [ 1601.596479] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.600623] LustreError: 50390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1601.911277] Lustre: server umount lustre-MDT0001 complete [ 1616.601677] Lustre: server umount lustre-OST0000 complete [ 1647.908035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.185106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1648.400703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.753729] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1661.048333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.136620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1661.168924] Lustre: 64265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1666.357638] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1674.350393] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.925739] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.606674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.878164] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1687.165392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.169700] Lustre: Skipped 1 previous similar message [ 1693.212354] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1701.263595] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1702.817848] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1702.830699] Lustre: Skipped 18 previous similar messages [ 1702.863116] Lustre: 66834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1714.149063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.168980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.181078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.743750] Lustre: server umount lustre-OST0000 complete [ 1725.336163] Lustre: server umount lustre-MDT0000 complete [ 1728.475243] LustreError: 64258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777837 with bad export cookie 3676151494179105041 [ 1728.484356] LustreError: 64258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1728.484603] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.919822] Lustre: server umount lustre-MDT0001 complete [ 1739.795174] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1742.886675] Key type lgssc unregistered [ 1743.330840] LNet: 68043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.337640] LNetError: 68043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.357884] LNet: Removed LNI 192.168.203.142@tcp [ 1744.380430] Key type .llcrypt unregistered [ 1744.385118] Key type ._llcrypt unregistered [ 1770.248182] Key type ._llcrypt registered [ 1770.253018] Key type .llcrypt registered [ 1770.355707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1771.669965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.704909] alg: No test for adler32 (adler32-zlib) [ 1772.745944] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 1772.976579] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1774.687171] Key type lgssc registered [ 1775.691833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.293038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.321164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.731527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.228199] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1795.042697] LustreError: 69455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.060952] LustreError: 69455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1796.662761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.224995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.426756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1810.465834] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.845357] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1824.104748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.739487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.401916] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1839.153882] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.645700] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1857.511778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.515048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.537713] Lustre: Skipped 1 previous similar message [ 1857.550172] Lustre: Skipped 1 previous similar message [ 1862.635560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.648416] Lustre: Skipped 3 previous similar messages [ 1863.604716] Lustre: server umount lustre-MDT0000 complete [ 1867.701646] LustreError: 69434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777976 with bad export cookie 17281345855427669383 [ 1867.705111] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1867.711607] LustreError: 69434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1867.745463] LustreError: 70703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.773283] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.789256] Lustre: Skipped 1 previous similar message [ 1867.794901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1868.021972] Lustre: server umount lustre-MDT0001 complete [ 1874.150091] Lustre: server umount lustre-OST0000 complete [ 1898.410978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.655464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1898.687193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1899.000482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.939493] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1910.392355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.501363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1910.518994] Lustre: 74321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1910.974731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.753156] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1922.170743] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.350583] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.095444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.263840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1933.652928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.176761] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1946.121314] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.507172] Lustre: 76898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1949.075703] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1956.835263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1956.847899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.865475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.714689] Lustre: server umount lustre-OST0000 complete [ 1967.075053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1967.082071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.099547] Lustre: Skipped 1 previous similar message [ 1967.107382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.114176] Lustre: Skipped 2 previous similar messages [ 1969.451885] Lustre: server umount lustre-MDT0000 complete [ 1972.705095] LustreError: 74334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.725200] LustreError: 74334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1973.221234] LustreError: 74315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778081 with bad export cookie 17281345855427670223 [ 1973.231591] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1973.242511] LustreError: 74315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1973.532544] Lustre: server umount lustre-MDT0001 complete [ 1983.795442] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1986.437864] Key type lgssc unregistered [ 1986.790915] LNet: 78106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.797723] LNetError: 78106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.817382] LNet: Removed LNI 192.168.203.142@tcp [ 1987.769792] Key type .llcrypt unregistered [ 1987.772855] Key type ._llcrypt unregistered [ 2013.922669] Key type ._llcrypt registered [ 2013.926907] Key type .llcrypt registered [ 2014.077279] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2015.319042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2015.399975] alg: No test for adler32 (adler32-zlib) [ 2016.796132] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 2017.168326] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2018.871239] Key type lgssc registered [ 2020.270737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2028.831043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2028.883549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.697710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.172377] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2040.814728] LustreError: 79515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.842991] LustreError: 79515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2045.941611] LustreError: 79516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.960597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.761747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.721471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2059.576358] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.737385] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.194644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.706717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.118960] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2087.640577] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2096.590887] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2097.540374] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.359325] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.363873] Lustre: Skipped 1 previous similar message [ 2101.982583] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2101.991659] Lustre: Skipped 2 previous similar messages [ 2106.210736] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2106.217358] Lustre: Skipped 4 previous similar messages [ 2114.224387] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2114.230144] Lustre: Skipped 9 previous similar messages [ 2130.715419] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2130.719773] Lustre: Skipped 20 previous similar messages [ 2163.060892] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2163.068468] Lustre: Skipped 31 previous similar messages [ 2197.132796] Lustre: 87622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2200.097242] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:38:26 (1789778306) [ 2201.962676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2203.914736] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:38:30 (1789778310) [ 2209.252754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2209.254302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.267974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.283239] Lustre: Skipped 1 previous similar message [ 2209.300393] Lustre: Skipped 1 previous similar message [ 2214.387069] LustreError: 81344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.407875] LustreError: 81344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2214.501545] Lustre: server umount lustre-OST0000 complete [ 2219.489743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.495204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.510554] Lustre: Skipped 1 previous similar message [ 2219.525036] Lustre: Skipped 1 previous similar message [ 2224.609582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.824580] Lustre: server umount lustre-MDT0000 complete [ 2228.331347] LustreError: 79497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778336 with bad export cookie 1444340568457494903 [ 2228.331675] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2228.340701] LustreError: 79497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2228.720811] Lustre: server umount lustre-MDT0001 complete [ 2239.379984] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2241.670519] Key type lgssc unregistered [ 2241.966251] LNet: 88869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.978271] LNetError: 88869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.998469] LNet: Removed LNI 192.168.203.142@tcp [ 2242.959920] Key type .llcrypt unregistered [ 2242.966512] Key type ._llcrypt unregistered [ 2247.044915] Key type ._llcrypt registered [ 2247.048153] Key type .llcrypt registered [ 2247.156899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2251.701905] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules [ 2252.859606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.887744] alg: No test for adler32 (adler32-zlib) [ 2253.897872] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 2254.109783] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2255.776211] Key type lgssc registered [ 2256.736306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.734619] LustreError: 90069:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2274.349117] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2279.925047] LustreError: 90069:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2282.986594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.002730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.443543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.199655] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2293.728405] LustreError: 91092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.758761] LustreError: 91092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2297.860351] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:40:04 (1789778404) [ 2298.852083] LustreError: 91093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.967044] LustreError: 91125:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2302.984826] LustreError: 91125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2302.996187] LustreError: 91125:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2303.360385] Lustre: server umount lustre-MDT0000 complete [ 2317.899849] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2320.658560] Key type lgssc unregistered [ 2320.880809] LNet: 92618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.888848] LNetError: 92618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.916614] LNet: Removed LNI 192.168.203.142@tcp [ 2321.616491] Key type .llcrypt unregistered [ 2321.619613] Key type ._llcrypt unregistered [ 2339.440836] Key type ._llcrypt registered [ 2339.443962] Key type .llcrypt registered [ 2339.545572] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2340.264307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.297084] alg: No test for adler32 (adler32-zlib) [ 2341.300863] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 2341.483020] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2343.151387] Key type lgssc registered [ 2344.109583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2351.697638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2351.719486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.614624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.538845] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2363.886729] LustreError: 93831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.909661] LustreError: 93831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2367.211487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.885035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2372.075777] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2379.228093] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.241683] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2392.131260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.934611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.239610] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2405.790318] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.438966] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:42:42 (1789778562) [ 2456.785208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2458.304627] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:42:45 (1789778565) [ 2464.737962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.747502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.782054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2465.769939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2465.771973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.125197] Lustre: server umount lustre-MDT0000 complete [ 2470.881522] LustreError: 93828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.898416] LustreError: 93828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2471.353344] LustreError: 93811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778579 with bad export cookie 10451035431047659033 [ 2471.356037] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.360650] LustreError: 93811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2471.646962] Lustre: server umount lustre-MDT0001 complete [ 2485.154311] Lustre: server umount lustre-OST0000 complete [ 2497.892634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.646093] LustreError: 98124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.861603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.801668] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2504.163313] LustreError: 98125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.282077] LustreError: 98124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.209502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.053704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2516.108928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2522.375210] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.560476] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2539.183774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.403281] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2539.414300] LustreError: 98113:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2539.463816] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2539.470740] LustreError: 100207:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2539.477645] LustreError: 100207:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2539.482747] LustreError: 100207:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2539.627379] Lustre: server umount lustre-OST0000 complete [ 2539.635479] LustreError: 100207:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2547.575555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.804863] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2547.810673] LustreError: 98114:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2547.841687] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2547.848578] LustreError: 100677:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2547.855588] LustreError: 100677:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2547.859411] LustreError: 100677:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2547.973572] Lustre: server umount lustre-OST0000 complete [ 2547.975852] LustreError: 100677:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2557.247156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.409462] LustreError: 101193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.487977] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2557.496354] Lustre: Found index 0 for lustre-OST0000, updating log [ 2557.505821] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2557.604720] Lustre: lustre-OST0000: new disk, initializing [ 2557.611868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2557.618274] Lustre: 101210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2558.010464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.936656] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2567.383933] Lustre: Failing over lustre-OST0000 [ 2567.495923] Lustre: server umount lustre-OST0000 complete [ 2573.280959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.291404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.298150] Lustre: Skipped 1 previous similar message [ 2573.304087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.310876] Lustre: Skipped 2 previous similar messages [ 2578.384866] Lustre: server umount lustre-MDT0000 complete [ 2578.920700] LustreError: 98124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.942096] LustreError: 98124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2582.715726] LustreError: 98106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778691 with bad export cookie 10451035431047661777 [ 2582.728126] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.735238] LustreError: 98106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.775397] LustreError: 98856:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2582.780918] LustreError: 98856:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2593.441562] Lustre: server umount lustre-MDT0001 complete [ 2603.067961] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:45:09 (1789778709) [ 2618.951377] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2629.519868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.912865] LustreError: 103557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.033898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.916138] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2643.807522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.802956] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2651.614904] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2658.257525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.012322] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2665.759147] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2669.222981] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2669.253862] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2669.254041] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2673.581450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.664429] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2673.759287] Lustre: lustre-OST0001: new disk, initializing [ 2673.764493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2673.771529] Lustre: 105878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2674.001084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.004647] Lustre: Skipped 2 previous similar messages [ 2679.451579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2679.465192] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2679.588096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2679.839628] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2687.544955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2691.582579] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2696.163357] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.170781] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.181143] Lustre: Skipped 1 previous similar message [ 2696.184567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2696.198969] Lustre: Skipped 1 previous similar message [ 2700.263169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2700.269603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.272072] Lustre: Skipped 1 previous similar message [ 2700.289543] Lustre: Skipped 1 previous similar message [ 2701.728562] Lustre: server umount lustre-MDT0001 complete [ 2703.094456] LustreError: 103553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.122285] LustreError: 103553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2711.884314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.992332] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2711.995667] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2712.025486] Lustre: 103547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2713.683339] LustreError: 107424:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2713.694518] LustreError: 107424:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2713.709296] LustreError: 103549:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2713.842174] LustreError: 107437:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -17 [ 2713.853494] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2713.853494] [ 2718.059500] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2718.189863] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2729.478819] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:47:16 (1789778836) [ 2746.588942] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2763.428357] Lustre: 109494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2785.991057] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2789.976578] Lustre: 110628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2881.504410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2881.509679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.521909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.525121] Lustre: Skipped 2 previous similar messages [ 2884.686117] Lustre: server umount lustre-MDT0000 complete [ 2887.137071] LustreError: 103553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.152948] LustreError: 103553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2889.223926] LustreError: 103537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778997 with bad export cookie 10451035431047662778 [ 2889.225773] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.234306] LustreError: 103537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2889.546559] Lustre: server umount lustre-MDT0001 complete [ 2904.222984] Lustre: server umount lustre-OST0000 complete [ 2908.114152] Lustre: 93141:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789779000/real 1789779000] req@ffff9fac40fcaa00 x1876718728341760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789779016 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2918.555559] Lustre: server umount lustre-OST0001 complete [ 2924.711529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2936.969058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2946.285957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2955.554837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2972.841748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2982.896464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.921994] Lustre: lustre-MDT0000: reset Object Index mappings [ 2983.475442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.490059] Lustre: Skipped 1 previous similar message [ 2987.624788] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2996.075096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.105951] Lustre: lustre-MDT0001: reset Object Index mappings [ 3001.524914] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3004.555249] Lustre: 115731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3010.389042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.499134] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3022.279054] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3024.463901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.157082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3028.160841] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3028.161542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3030.528455] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3038.084350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3041.764477] Lustre: 117609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3091.508856] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:53:18 (1789779198) [ 3119.587715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3119.599358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.611050] Lustre: Skipped 3 previous similar messages [ 3119.615670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.626044] Lustre: Skipped 4 previous similar messages [ 3124.693278] Lustre: server umount lustre-MDT0000 complete [ 3128.092199] LustreError: 115332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779236 with bad export cookie 10451035431047709629 [ 3128.102057] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.111201] LustreError: 115332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3134.680648] Lustre: server umount lustre-MDT0001 complete [ 3144.499752] Lustre: server umount lustre-OST0000 complete [ 3148.088985] LustreError: 118573:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x73:0x0].0x0 (ffff9fab0fa7b200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3154.387405] Lustre: server umount lustre-OST0001 complete [ 3161.122951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3172.193743] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3210.465526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.673837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3210.699168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3210.840308] Lustre: lustre-MDT0000: new disk, initializing [ 3211.036277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.042675] Lustre: Skipped 3 previous similar messages [ 3211.056730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3215.706965] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3223.022381] Lustre: server umount lustre-MDT0000 complete [ 3231.979336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.228687] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3232.253557] Lustre: 121882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3232.268753] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3236.137738] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3240.593732] Lustre: server umount lustre-MDT0000 complete [ 3249.862428] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:55:56 (1789779356) [ 3251.834496] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3253.759709] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:56:00 (1789779360) [ 3272.915328] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3284.030828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.904279] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3298.516756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.607661] Lustre: 123932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3298.656810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3298.661476] Lustre: Skipped 1 previous similar message [ 3298.738720] Lustre: lustre-MDT0001: new disk, initializing [ 3298.848214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3298.871353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3303.820694] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3308.556727] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3314.862789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.072874] Lustre: lustre-OST0000: new disk, initializing [ 3315.075644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3315.080696] Lustre: 125510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3316.224748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3316.240283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3316.309616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3321.108428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3331.330949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.500461] Lustre: lustre-OST0001: new disk, initializing [ 3331.506742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3331.518261] Lustre: 126380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3332.792206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3332.806989] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3332.918323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3338.185344] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3347.509987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3350.904497] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3719.055144] Lustre: 132807:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3719.070306] Lustre: 132807:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3719.077805] Lustre: 132807:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3719.092187] Lustre: 132807:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3719.104275] Lustre: 132807:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3719.111878] Lustre: 132807:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3985.517937] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:08:11 (1789780091) [ 4001.251330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.264653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.276570] Lustre: Skipped 5 previous similar messages [ 4001.283837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.288476] Lustre: Skipped 8 previous similar messages [ 4006.016363] Lustre: server umount lustre-MDT0000 complete [ 4008.931165] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4008.960275] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4009.961883] LustreError: 132855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789780118 with bad export cookie 10451035431047778250 [ 4009.976242] LustreError: 132855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4009.981278] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.058821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.075740] Lustre: Skipped 3 previous similar messages [ 4014.092176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4014.097884] Lustre: Skipped 5 previous similar messages [ 4016.534538] Lustre: server umount lustre-MDT0001 complete [ 4027.405956] Lustre: server umount lustre-OST0000 complete [ 4037.385440] Lustre: server umount lustre-OST0001 complete [ 4044.346495] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4055.383971] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4091.444859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.740822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4091.787429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4091.903520] Lustre: lustre-MDT0000: new disk, initializing [ 4091.975495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.984181] Lustre: Skipped 5 previous similar messages [ 4091.995705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4096.502161] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4106.598086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.751175] Lustre: 135857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4106.800556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4106.804433] Lustre: Skipped 1 previous similar message [ 4106.874434] Lustre: lustre-MDT0001: new disk, initializing [ 4106.999656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4107.016794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.847803] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4121.312081] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4127.463311] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4134.051078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.421613] Lustre: lustre-OST0000: new disk, initializing [ 4134.427851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4134.439541] Lustre: 137825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4134.538678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.545710] Lustre: Skipped 1 previous similar message [ 4136.141328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4136.151245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4136.291548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4140.366758] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4148.953138] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4157.832667] Lustre: lustre-MDD0000: changelog on [ 4160.148855] Lustre: lustre-MDD0001: changelog on [ 6248.481692] Lustre: lustre-MDD0001: changelog off [ 6250.458240] Lustre: 135892:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6251.465762] Lustre: lustre-MDD0000: changelog off [ 6261.217946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6261.219757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.233710] LustreError: Skipped 1 previous similar message [ 6261.279834] Lustre: Skipped 2 previous similar messages [ 6261.286213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6261.300445] Lustre: Skipped 1 previous similar message [ 6264.763799] Lustre: server umount lustre-OST0000 complete [ 6271.972803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.997666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6272.005816] Lustre: Skipped 1 previous similar message [ 6277.090401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6277.099314] Lustre: Skipped 1 previous similar message [ 6282.211823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.225622] Lustre: Skipped 1 previous similar message [ 6283.231134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6283.472258] Lustre: server umount lustre-MDT0000 complete [ 6287.171909] LustreError: 135848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782395 with bad export cookie 10451035431048559415 [ 6287.180358] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6287.181924] LustreError: 135848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6287.605801] Lustre: server umount lustre-MDT0001 complete [ 6300.079430] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6303.543580] Key type lgssc unregistered [ 6303.814350] LNet: 147006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6303.831663] LNetError: 147006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6303.849997] LNet: Removed LNI 192.168.203.142@tcp [ 6304.965175] Key type .llcrypt unregistered [ 6304.966921] Key type ._llcrypt unregistered [ 6327.723934] Key type ._llcrypt registered [ 6327.730236] Key type .llcrypt registered [ 6327.961312] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6349.278089] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6350.546176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6350.720750] alg: No test for adler32 (adler32-zlib) [ 6351.749683] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 6352.019217] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6353.727157] Key type lgssc registered [ 6354.918410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6394.205685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6394.234314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.535210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6395.575992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6395.677357] Lustre: lustre-MDT0000: new disk, initializing [ 6395.794844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.823777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6401.748283] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6416.145533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.289321] Lustre: 150220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6416.392253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6416.398543] Lustre: Skipped 1 previous similar message [ 6416.553957] Lustre: lustre-MDT0001: new disk, initializing [ 6416.700605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6416.766307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6416.782868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6422.674824] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6433.813580] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6440.911991] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6449.653328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.949304] Lustre: lustre-OST0000: new disk, initializing [ 6449.958616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6449.965231] Lustre: 152190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6450.080238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.319541] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6458.505801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6458.532518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6458.765570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6469.749717] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6475.708367] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module obdecho/obdecho [ 6477.914853] LustreError: 153194:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6477.924616] LustreError: 153194:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6479.835181] LustreError: 153290:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6493.664616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6493.680797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.716467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6494.696311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6494.720279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.743891] Lustre: Skipped 1 previous similar message [ 6498.791068] Lustre: server umount lustre-MDT0000 complete [ 6503.357537] LustreError: 152189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782611 with bad export cookie 13101795620145468541 [ 6503.362417] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.365156] LustreError: 152189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6503.839756] Lustre: server umount lustre-MDT0001 complete [ 6519.369575] Lustre: server umount lustre-OST0000 complete [ 6545.529909] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6557.633634] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6601.546454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.774652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6601.819453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6601.893528] Lustre: lustre-MDT0000: new disk, initializing [ 6601.964310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.976884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6607.626185] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6619.524943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.691236] Lustre: 157010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6619.753036] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6619.758129] Lustre: Skipped 1 previous similar message [ 6619.886923] Lustre: lustre-MDT0001: new disk, initializing [ 6620.023303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6620.078315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6620.103513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6625.039648] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6635.017464] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6641.802734] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6649.514110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6649.878699] Lustre: lustre-OST0000: new disk, initializing [ 6649.886895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6649.897886] Lustre: 158978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6650.010911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.357421] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6658.686546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6658.712130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6658.895517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6667.804654] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6672.453789] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6672.703769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6677.915137] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6678.225509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6684.648051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6684.651804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.661946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.671482] Lustre: Skipped 1 previous similar message [ 6684.696500] Lustre: Skipped 3 previous similar messages [ 6686.568575] Lustre: server umount lustre-OST0000 complete [ 6691.808628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6691.826203] LustreError: Skipped 1 previous similar message [ 6691.827979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.856989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.001440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.011969] Lustre: Skipped 3 previous similar messages [ 6705.126330] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6705.477754] Lustre: server umount lustre-MDT0000 complete [ 6709.922813] LustreError: 157002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782818 with bad export cookie 13101795620145469500 [ 6709.928155] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.937414] LustreError: 157002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6710.380941] Lustre: server umount lustre-MDT0001 complete [ 6714.602519] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:53:41 (1789782821) [ 6716.761720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6719.143087] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:53:45 (1789782825) [ 6727.626228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.222505] LustreError: 161394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.240561] LustreError: 161394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6728.308329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.241708] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6733.793838] LustreError: 161395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.888035] LustreError: 161394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.815750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.019310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6743.031317] Lustre: Skipped 2 previous similar messages [ 6749.492699] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6758.622987] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6765.061276] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6772.944215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.256161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.269138] Lustre: Skipped 1 previous similar message [ 6779.664871] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6789.381560] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.798874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.816877] Lustre: Skipped 1 previous similar message [ 6797.826417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.841681] Lustre: Skipped 1 previous similar message [ 6802.349423] Lustre: server umount lustre-MDT0000 complete [ 6802.916315] LustreError: 162334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.960554] LustreError: 162334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6806.973901] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782915 with bad export cookie 13101795620145470375 [ 6806.995916] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.032481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.032792] LustreError: 161389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.043962] Lustre: Skipped 2 previous similar messages [ 6808.063605] LustreError: 161389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6816.236795] LustreError: 161391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.258399] LustreError: 161391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6821.344374] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6821.800501] Lustre: server umount lustre-MDT0001 complete [ 6829.779280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6843.532190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.783506] LustreError: 165234:0:(llog.c:1655:llog_backup()) MGC192.168.203.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6843.790871] Lustre: 165234:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6846.945104] LustreError: 165240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.959555] LustreError: 165240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6881.761866] LustreError: 165241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6881.786088] LustreError: 165241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 6904.415493] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6904.423148] Lustre: 165234:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6945.286143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.064792] LustreError: 165239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.082499] LustreError: 165239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6950.027667] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6950.406272] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6959.818057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.878620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6983.591221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6983.673638] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6992.791018] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6995.256149] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7003.766678] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7006.171670] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7029.733958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.740392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.751497] Lustre: Skipped 2 previous similar messages [ 7029.760839] Lustre: Skipped 10 previous similar messages [ 7034.849606] Lustre: server umount lustre-MDT0000 complete [ 7038.434935] LustreError: 161376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783147 with bad export cookie 13101795620145476871 [ 7038.442263] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.449229] LustreError: 161376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7038.706447] Lustre: server umount lustre-MDT0001 complete [ 7053.294221] Lustre: server umount lustre-OST0000 complete [ 7074.361419] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7084.690965] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7122.740808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.931680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7122.950700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7123.002461] Lustre: lustre-MDT0000: new disk, initializing [ 7123.067292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.070463] Lustre: Skipped 1 previous similar message [ 7123.080219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7127.714562] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7138.791904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.883240] Lustre: 170347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7139.161194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7143.976206] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7154.329651] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7159.922971] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7167.941560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.157609] Lustre: lustre-OST0000: new disk, initializing [ 7168.164102] Lustre: Skipped 1 previous similar message [ 7168.170306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7168.177105] Lustre: Skipped 2 previous similar messages [ 7168.182282] Lustre: 172329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7174.713205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7174.716986] Lustre: Skipped 1 previous similar message [ 7174.720089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7174.816300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7175.116197] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7186.170859] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7192.983904] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7193.264399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7198.414826] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7198.646990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7200.742467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.744079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7200.758557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7200.764566] LustreError: Skipped 1 previous similar message [ 7200.780341] Lustre: Skipped 1 previous similar message [ 7206.764159] Lustre: server umount lustre-OST0000 complete [ 7210.979171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7217.051569] Lustre: server umount lustre-MDT0000 complete [ 7221.220899] LustreError: 171190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.262333] LustreError: 171190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 7221.428365] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783330 with bad export cookie 13101795620145493657 [ 7221.437287] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.440908] LustreError: 170339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7222.017325] Lustre: server umount lustre-MDT0001 complete [ 7231.980833] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:02:18 (1789783338) [ 7239.414701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.813891] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7254.371338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.635942] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7267.309529] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7273.350836] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7280.368573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7287.895843] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7296.553064] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7310.334636] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:03:36 (1789783416) [ 7314.921583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7314.925188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.954918] Lustre: Skipped 4 previous similar messages [ 7321.031551] Lustre: server umount lustre-OST0000 complete [ 7330.275148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.281791] Lustre: Skipped 13 previous similar messages [ 7331.264753] Lustre: server umount lustre-MDT0000 complete [ 7335.431915] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783444 with bad export cookie 13101795620145494539 [ 7335.443342] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.453455] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7335.936824] Lustre: server umount lustre-MDT0001 complete [ 7348.543238] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7351.781293] Key type lgssc unregistered [ 7352.132728] LNet: 178470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.145750] LNetError: 178470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.163137] LNet: Removed LNI 192.168.203.142@tcp [ 7353.153277] Key type .llcrypt unregistered [ 7353.155842] Key type ._llcrypt unregistered [ 7375.531693] Key type ._llcrypt registered [ 7375.534626] Key type .llcrypt registered [ 7375.625695] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7376.948992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.066983] alg: No test for adler32 (adler32-zlib) [ 7378.139167] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 7378.490948] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7380.239218] Key type lgssc registered [ 7381.329039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7389.358936] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7389.381563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.023389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7410.143510] LustreError: 179714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 7415.273575] LustreError: 179714:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7415.297618] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7415.307719] LustreError: MGC192.168.203.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7415.330897] LustreError: 179714:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7415.347625] LustreError: 179714:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7415.367884] LustreError: 179714:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7415.436965] Lustre: server umount lustre-OST0000 complete [ 7415.438949] LustreError: 179714:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7424.701892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.546079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7430.297286] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7436.770188] LustreError: 180215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.790505] LustreError: 180215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7439.538541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.960213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7444.616485] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7453.249811] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.606670] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.401390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.694764] LustreError: 182031:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7467.717544] Lustre: 182031:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7467.923441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.356942] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7482.566510] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.752210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.757311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.779080] Lustre: Skipped 2 previous similar messages [ 7485.780375] Lustre: Skipped 1 previous similar message [ 7491.045768] LustreError: 181218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.060167] LustreError: 181218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7491.252714] Lustre: server umount lustre-MDT0000 complete [ 7495.888044] LustreError: 182058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783604 with bad export cookie 3667459492959838918 [ 7495.890088] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7495.911946] LustreError: 182058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7496.095521] LustreError: 180943:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d349df46 x1876724009693696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7496.160208] LustreError: 180210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.163266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.207545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7496.217402] Lustre: Skipped 1 previous similar message [ 7496.740514] Lustre: server umount lustre-MDT0001 complete [ 7503.032387] Lustre: server umount lustre-OST0000 complete [ 7512.747797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7523.395347] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7558.375705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.588989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7558.612127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7558.686300] Lustre: lustre-MDT0000: new disk, initializing [ 7558.759959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7558.776257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7562.768977] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7573.350957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.517437] Lustre: 185553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7573.588341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7573.593171] Lustre: Skipped 1 previous similar message [ 7573.668276] Lustre: lustre-MDT0001: new disk, initializing [ 7573.738307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.787896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7573.811526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7578.263810] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7588.195494] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7594.992726] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.982550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.173374] Lustre: lustre-OST0000: new disk, initializing [ 7602.178308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7602.187773] Lustre: 187522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7602.322685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.982352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7607.996241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7608.256050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7611.177503] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7623.683573] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.316131] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7629.569633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7636.143714] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7636.455387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7644.130852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.136678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.151174] LustreError: Skipped 1 previous similar message [ 7644.183431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.217603] Lustre: Skipped 1 previous similar message [ 7645.747973] Lustre: server umount lustre-OST0000 complete [ 7654.375943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.394284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.394771] Lustre: Skipped 2 previous similar messages [ 7657.293861] Lustre: server umount lustre-MDT0000 complete [ 7659.494164] LustreError: 186565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.516682] LustreError: 186565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7661.220732] LustreError: 186348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783769 with bad export cookie 3667459492959839639 [ 7661.230248] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7661.241062] LustreError: 186348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7661.626627] Lustre: server umount lustre-MDT0001 complete [ 7671.208314] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:09:38 (1789783778) [ 7689.798125] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7692.491561] Key type lgssc unregistered [ 7692.756069] LNet: 190469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7692.781766] LNetError: 190469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7692.808462] LNet: Removed LNI 192.168.203.142@tcp [ 7693.673178] Key type .llcrypt unregistered [ 7693.676194] Key type ._llcrypt unregistered [ 7712.610615] Key type ._llcrypt registered [ 7712.612753] Key type .llcrypt registered [ 7712.776166] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7713.790039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7713.840203] alg: No test for adler32 (adler32-zlib) [ 7715.057187] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 7715.355088] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7717.079169] Key type lgssc registered [ 7718.247172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7726.571565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7726.585032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.015855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.188870] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7738.344597] LustreError: 191684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.384941] LustreError: 191684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7741.982913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.407581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.433311] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7754.606455] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7761.572634] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7771.526942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.860778] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7771.865324] LustreError: 193550:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7771.871765] Lustre: 193550:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7771.984806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.207824] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7788.629736] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.617128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.629928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7792.632222] Lustre: Skipped 1 previous similar message [ 7797.729200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.739858] Lustre: Skipped 2 previous similar messages [ 7802.849173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.852023] Lustre: Skipped 1 previous similar message [ 7805.407143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7805.600502] Lustre: server umount lustre-MDT0000 complete [ 7807.991525] LustreError: 191684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7808.001088] LustreError: 191684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7809.727117] LustreError: 191665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783918 with bad export cookie 4838515094089497981 [ 7809.729241] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.735382] LustreError: 191665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7810.423555] Lustre: server umount lustre-MDT0001 complete [ 7824.762896] Lustre: server umount lustre-OST0000 complete [ 7833.283815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7844.830437] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7882.743187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.004128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7883.037687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7883.126705] Lustre: lustre-MDT0000: new disk, initializing [ 7883.173748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.194421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7888.058459] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7897.655950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.778309] Lustre: 197079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7897.843287] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7897.846439] Lustre: Skipped 1 previous similar message [ 7897.853657] LustreError: 197861:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7897.862955] LustreError: 197861:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7897.871228] Lustre: 197861:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7897.895200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7897.899808] Lustre: Skipped 1 previous similar message [ 7897.963324] Lustre: lustre-MDT0001: new disk, initializing [ 7898.064353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7898.110150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7898.122322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7902.651646] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7913.116894] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7919.753234] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.120846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.467676] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7927.477475] Lustre: Skipped 1 previous similar message [ 7927.482725] LustreError: 199023:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7927.492687] LustreError: 199023:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7927.505702] Lustre: 199023:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7927.530899] Lustre: 199023:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7927.567780] Lustre: lustre-OST0000: new disk, initializing [ 7927.578911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7927.594964] Lustre: 199046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7927.672510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.024527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7934.036722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7934.143198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7934.815839] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7945.042424] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7949.275521] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7949.502769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7954.159814] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7954.396943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7960.035798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.041602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7960.043820] Lustre: Skipped 2 previous similar messages [ 7960.045544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.045549] Lustre: Skipped 1 previous similar message [ 7962.378212] Lustre: server umount lustre-OST0000 complete [ 7969.760562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7969.773599] LustreError: Skipped 1 previous similar message [ 7969.776254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7969.796940] Lustre: Skipped 1 previous similar message [ 7969.804504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.818016] Lustre: Skipped 1 previous similar message [ 7972.493616] Lustre: server umount lustre-MDT0000 complete [ 7975.392510] LustreError: 197090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7975.412930] LustreError: 197090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7976.269876] LustreError: 197070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784084 with bad export cookie 4838515094089498702 [ 7976.287030] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7976.292038] LustreError: 197070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7976.775123] Lustre: server umount lustre-MDT0001 complete [ 7985.981829] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:14:52 (1789784092) [ 8004.347100] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8016.423827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.948734] LustreError: 202177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.002449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.282349] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8022.506893] LustreError: 202178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.554968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.715627] LustreError: 202177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.732247] LustreError: 202177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8036.864943] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8040.391750] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8046.667521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.285558] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8062.401814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.609646] Lustre: lustre-OST0001: new disk, initializing [ 8062.613886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8062.621781] Lustre: 204483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8062.719323] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8062.726816] Lustre: Skipped 2 previous similar messages [ 8070.644319] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8071.757451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8071.766167] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8071.943044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8082.176030] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8090.494659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8092.698527] Lustre: DEBUG MARKER: ADD OST3 [ 8097.686359] loop: module loaded [ 8097.810205] loop0: detected capacity change from 0 to 204800000 [ 8098.122625] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8098.138204] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8098.157638] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8105.453472] Lustre: DEBUG MARKER: STOP OST3 [ 8105.482353] loop0: detected capacity change from 0 to 204800000 [ 8112.153541] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8112.474486] Lustre: lustre-OST0003: new disk, initializing [ 8112.481298] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8112.497989] Lustre: 206533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.577559] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8113.772813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8113.783243] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8113.826235] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8115.490333] Lustre: Failing over lustre-OST0003 [ 8115.618394] LustreError: 206622:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8115.740090] Lustre: server umount lustre-OST0003 complete [ 8118.773265] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8118.795062] LustreError: 203662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8118.797541] Lustre: Skipped 2 previous similar messages [ 8118.802738] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8118.821200] LustreError: 203662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8119.880580] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8127.892915] Lustre: DEBUG MARKER: START OST3 again [ 8135.867387] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8136.242028] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8136.251684] Lustre: Skipped 1 previous similar message [ 8136.263760] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8137.626239] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8137.880115] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8137.881144] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8145.891625] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8167.324649] Lustre: Failing over lustre-OST0003 [ 8167.491945] Lustre: server umount lustre-OST0003 complete [ 8168.933189] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8168.936434] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.943168] LustreError: Skipped 1 previous similar message [ 8168.944090] LustreError: 204476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.944098] LustreError: 204476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8169.031857] Lustre: Skipped 1 previous similar message [ 8185.824920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8185.829722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.838283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.841693] Lustre: Skipped 2 previous similar messages [ 8191.829588] Lustre: server umount lustre-MDT0000 complete [ 8195.527801] LustreError: 202157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784304 with bad export cookie 4838515094089499584 [ 8195.557185] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8201.933744] Lustre: server umount lustre-MDT0001 complete [ 8206.585617] Lustre: server umount lustre-OST0000 complete [ 8211.075725] Lustre: server umount lustre-OST0001 complete [ 8220.272237] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8229.423798] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8268.009053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.341167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8268.393426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8268.518252] Lustre: lustre-MDT0000: new disk, initializing [ 8268.629506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.655333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8273.130823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8285.859931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.006374] Lustre: 211945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8286.075213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8286.079652] Lustre: Skipped 1 previous similar message [ 8286.141436] Lustre: lustre-MDT0001: new disk, initializing [ 8286.279802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8286.297477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8291.344648] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8301.225428] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8307.563294] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8314.091821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.418585] Lustre: 213913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8315.542752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8315.710072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8320.865755] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8329.981892] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8336.831732] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8337.209570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8345.121703] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8345.409460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.717901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.727544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.737510] LustreError: Skipped 1 previous similar message [ 8351.738051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.750736] Lustre: Skipped 6 previous similar messages [ 8351.753982] Lustre: Skipped 8 previous similar messages [ 8353.773171] Lustre: server umount lustre-OST0000 complete [ 8364.391426] Lustre: server umount lustre-MDT0000 complete [ 8367.072383] LustreError: 213058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.104114] LustreError: 213058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 8369.423381] LustreError: 211937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784478 with bad export cookie 4838515094089501880 [ 8369.438949] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8369.449048] LustreError: 211937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8369.813554] Lustre: server umount lustre-MDT0001 complete [ 8373.981137] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:21:20 (1789784480) [ 8391.664531] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8402.591523] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8444.919253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.124792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8445.146629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8445.150664] Lustre: Skipped 1 previous similar message [ 8445.236422] Lustre: lustre-MDT0000: new disk, initializing [ 8445.238735] Lustre: Skipped 1 previous similar message [ 8445.299093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8445.304034] Lustre: Skipped 1 previous similar message [ 8449.369081] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8460.631595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.793764] Lustre: 218523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8461.143596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8466.695508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8477.364784] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8484.630274] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8491.683449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.975899] Lustre: 220494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8493.734378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8493.855371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8498.941301] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8509.091962] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8513.911594] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8514.168142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8518.511445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8518.753524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8524.777039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8524.778711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.790177] LustreError: Skipped 1 previous similar message [ 8524.811253] Lustre: Skipped 2 previous similar messages [ 8524.822502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8524.829651] Lustre: Skipped 5 previous similar messages [ 8526.825269] Lustre: server umount lustre-OST0000 complete [ 8537.640627] Lustre: server umount lustre-MDT0000 complete [ 8540.129336] LustreError: 218533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8540.137653] LustreError: 218533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8541.286101] LustreError: 219320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784649 with bad export cookie 4838515094089502762 [ 8541.294905] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8541.300440] LustreError: 219320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8541.640153] Lustre: server umount lustre-MDT0001 complete [ 8550.197701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.786988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.791166] Lustre: Skipped 5 previous similar messages [ 8555.364498] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8566.749564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.226745] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8583.389730] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.086927] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8597.648917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8604.103167] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8611.434930] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8632.800946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8632.820991] LustreError: Skipped 2 previous similar messages [ 8632.823062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.871166] Lustre: Skipped 3 previous similar messages [ 8632.877494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8632.880362] Lustre: Skipped 3 previous similar messages [ 8638.658489] Lustre: server umount lustre-OST0000 complete [ 8656.351988] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8656.649662] Lustre: server umount lustre-MDT0000 complete [ 8661.520381] LustreError: 224557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784770 with bad export cookie 4838515094089503644 [ 8661.534951] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8661.535831] LustreError: 224557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8661.884544] Lustre: server umount lustre-MDT0001 complete [ 8751.957822] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8754.751845] Key type lgssc unregistered [ 8755.038847] LNet: 226399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8755.044701] LNetError: 226399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8755.066987] LNet: Removed LNI 192.168.203.142@tcp [ 8756.175676] Key type .llcrypt unregistered [ 8756.179899] Key type ._llcrypt unregistered [ 8783.898645] Key type ._llcrypt registered [ 8783.902757] Key type .llcrypt registered [ 8784.050554] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8806.843703] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8808.338398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8808.417448] alg: No test for adler32 (adler32-zlib) [ 8809.710314] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 8809.968444] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 8811.671461] Key type lgssc registered [ 8813.031220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.361203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.400914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.674792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8850.713294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8850.808331] Lustre: lustre-MDT0000: new disk, initializing [ 8850.864458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8850.882417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8855.440922] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8865.833599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.971628] Lustre: 229749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8866.017950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8866.022140] Lustre: Skipped 1 previous similar message [ 8866.124586] Lustre: lustre-MDT0001: new disk, initializing [ 8866.273851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.360375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8866.376678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8869.897380] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8880.818641] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8887.708260] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8895.592504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8895.932268] Lustre: lustre-OST0000: new disk, initializing [ 8895.940565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8895.952535] Lustre: 231718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8896.075595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.208400] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8906.294819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8906.309331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8906.412765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8913.896417] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8919.481816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8919.699573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8924.794701] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8925.044670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8927.205741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8927.216307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.221963] LustreError: Skipped 1 previous similar message [ 8927.222995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.247928] Lustre: Skipped 1 previous similar message [ 8932.331786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8932.344154] Lustre: Skipped 2 previous similar messages [ 8933.181474] Lustre: server umount lustre-OST0000 complete [ 8937.958277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8937.979170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.015206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.560192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.561765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.964462] Lustre: server umount lustre-MDT0000 complete [ 8947.822741] LustreError: 230545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785056 with bad export cookie 13758144562651150894 [ 8947.834141] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.836116] LustreError: 230545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8948.337654] Lustre: server umount lustre-MDT0001 complete [ 8967.164815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8976.870925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.354547] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.368681] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8977.412711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.508627] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8982.498096] LustreError: 234655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8986.597189] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.602780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.988092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8994.199251] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8997.023605] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9002.759332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.127144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.545751] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9016.834733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.081769] Lustre: lustre-OST0001: new disk, initializing [ 9017.090690] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9017.106539] Lustre: 236960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9025.244975] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9027.224471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9027.247933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9027.345710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9038.447500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9046.744077] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9061.677098] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:32:48 (1789785168) [ 9066.979486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9066.998395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.030141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9067.038330] Lustre: Skipped 1 previous similar message [ 9069.952913] Lustre: server umount lustre-MDT0000 complete [ 9073.612038] LustreError: 234634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785182 with bad export cookie 13758144562651151776 [ 9073.616200] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9073.626263] LustreError: 234634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9073.653470] LustreError: 235962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.657246] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.670353] LustreError: 235962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9073.689792] Lustre: Skipped 3 previous similar messages [ 9073.887954] Lustre: server umount lustre-MDT0001 complete [ 9080.003831] Lustre: server umount lustre-OST0000 complete [ 9084.130722] Lustre: server umount lustre-OST0001 complete [ 9091.803145] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 9102.242260] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9139.223955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.397963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9139.434053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9139.504787] Lustre: lustre-MDT0000: new disk, initializing [ 9139.560444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9139.567728] Lustre: Skipped 1 previous similar message [ 9139.586332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9144.253877] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9154.527743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.684989] Lustre: 240900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9154.757241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9154.761845] Lustre: Skipped 1 previous similar message [ 9154.854208] Lustre: lustre-MDT0001: new disk, initializing [ 9154.990064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9155.009213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9160.538931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9171.156950] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.150895] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9183.186722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.405513] Lustre: lustre-OST0000: new disk, initializing [ 9183.408902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9183.417954] Lustre: 242871:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9183.536838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.539818] Lustre: Skipped 1 previous similar message [ 9185.456694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9185.464663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9185.733776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9192.071585] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9202.169577] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9207.130082] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9207.366039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9212.196167] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9212.505247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9216.480339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9216.487507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9216.490184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9216.516254] Lustre: Skipped 2 previous similar messages [ 9216.521802] Lustre: Skipped 6 previous similar messages [ 9229.279539] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9229.431754] Lustre: server umount lustre-OST0000 complete [ 9236.960503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9236.963375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.970629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.970638] Lustre: Skipped 4 previous similar messages [ 9236.975596] LustreError: Skipped 1 previous similar message [ 9236.990584] Lustre: Skipped 1 previous similar message [ 9247.199990] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9247.400749] Lustre: server umount lustre-MDT0000 complete [ 9250.869735] LustreError: 240895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785359 with bad export cookie 13758144562651153911 [ 9250.874539] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.886042] LustreError: 240895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9251.428586] Lustre: server umount lustre-MDT0001 complete [ 9260.153881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.921754] LustreError: 245090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9261.010486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.806740] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9271.271202] LustreError: 245091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.287599] LustreError: 245091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9276.864274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.773675] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9289.211529] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9294.547705] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9303.481930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9309.664230] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9317.487971] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9465.831677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.842843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9465.867594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9465.891525] Lustre: Skipped 6 previous similar messages [ 9471.371189] Lustre: server umount lustre-OST0000 complete [ 9489.375397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9489.691558] Lustre: server umount lustre-MDT0000 complete [ 9491.425812] LustreError: 245090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.444550] LustreError: 245090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9493.560543] LustreError: 245070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785602 with bad export cookie 13758144562651154786 [ 9493.584352] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9493.589748] LustreError: 245070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9493.913620] Lustre: server umount lustre-MDT0001 complete [ 9587.412453] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 9590.989248] Key type lgssc unregistered [ 9591.557884] LNet: 248700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9591.570056] LNetError: 248700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9591.601439] LNet: Removed LNI 192.168.203.142@tcp [ 9592.809206] Key type .llcrypt unregistered [ 9592.813482] Key type ._llcrypt unregistered [ 9623.960736] Key type ._llcrypt registered [ 9623.964226] Key type .llcrypt registered [ 9624.217587] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 9641.190574] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9642.665571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9642.755498] alg: No test for adler32 (adler32-zlib) [ 9644.085965] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 9644.348183] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 9645.999212] Key type lgssc registered [ 9647.578711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9698.122876] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9708.090877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9708.127637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.401338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9709.449027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9709.537372] Lustre: lustre-MDT0000: new disk, initializing [ 9709.629678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.646672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9714.113599] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9725.807893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.954532] Lustre: 252773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9726.006318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9726.013938] Lustre: Skipped 1 previous similar message [ 9726.116314] Lustre: lustre-MDT0001: new disk, initializing [ 9726.232470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.303189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9726.330068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9732.094227] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9737.705052] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9745.244700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9745.500990] Lustre: lustre-OST0000: new disk, initializing [ 9745.505393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9745.512035] Lustre: 254404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9745.647124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.186345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9752.196994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9752.408693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9753.317521] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9764.460590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.660743] Lustre: lustre-OST0001: new disk, initializing [ 9764.665590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9764.672861] Lustre: 255276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9764.781059] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9771.103063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9771.118206] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9771.338659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9771.649254] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9782.972505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9791.463871] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9807.348115] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.355305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9807.367696] Lustre: Skipped 3 previous similar messages [ 9807.390802] Lustre: Skipped 3 previous similar messages [ 9810.246443] Lustre: server umount lustre-MDT0000 complete [ 9814.653176] LustreError: 252765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785923 with bad export cookie 14541553741508148569 [ 9814.662644] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.666336] LustreError: 252765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9815.166171] Lustre: server umount lustre-MDT0001 complete [ 9829.964854] Lustre: server umount lustre-OST0000 complete [ 9844.514883] Lustre: server umount lustre-OST0001 complete [ 9861.564945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9872.757767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9881.758214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9892.045124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9914.171055] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9924.809909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.826930] Lustre: lustre-MDT0000: reset Object Index mappings [ 9925.229174] LustreError: 260288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.240681] LustreError: 260288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9925.306996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9930.448782] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9930.733443] LustreError: 260289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.829518] LustreError: 260288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.409749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.423931] Lustre: lustre-MDT0001: reset Object Index mappings [ 9939.840296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9944.424889] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9947.140042] Lustre: 261430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9954.285070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.222198] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9965.093505] LustreError: 261786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.153136] LustreError: 261786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.203136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9970.447986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.456610] Lustre: Skipped 1 previous similar message [ 9971.514085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9971.521772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9976.863359] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9985.051937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9989.620088] Lustre: 263302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9996.258851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9996.273892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.316038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.284746] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.304669] Lustre: Skipped 2 previous similar messages [10001.837193] Lustre: server umount lustre-MDT0000 complete [10002.404385] LustreError: 260285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.434462] LustreError: 260285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10006.185401] LustreError: 260270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786114 with bad export cookie 14541553741508155366 [10006.192364] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.205715] LustreError: 260270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10006.715945] Lustre: server umount lustre-MDT0001 complete [10021.849896] Lustre: server umount lustre-OST0000 complete [10023.903389] Lustre: 250278:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789786116/real 1789786116] req@ffff9fab037d0700 x1876726384819328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789786132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10023.920715] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.614748] Lustre: server umount lustre-OST0001 complete [10049.533235] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:49:15 (1789786155) [10068.271549] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [10080.057359] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10123.340652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.796344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10123.853482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10124.069438] Lustre: lustre-MDT0000: new disk, initializing [10124.270375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10124.344900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10130.341613] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10142.785626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.039652] Lustre: 267560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10143.171261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10143.192083] Lustre: Skipped 1 previous similar message [10143.357970] Lustre: lustre-MDT0001: new disk, initializing [10143.708058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10143.734755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10149.734962] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10161.654104] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10168.561800] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10176.329098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10176.647601] Lustre: lustre-OST0000: new disk, initializing [10176.652081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10176.665152] Lustre: 269529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10178.559923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10178.570117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10178.808191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10184.048715] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10194.869702] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10200.409946] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10200.638130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10206.185717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10206.458762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10209.250834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10209.256900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10209.262258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10209.262266] Lustre: Skipped 4 previous similar messages [10209.267835] Lustre: Skipped 1 previous similar message [10214.384486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10214.398410] Lustre: Skipped 2 previous similar messages [10214.886567] Lustre: server umount lustre-OST0000 complete [10219.489700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.502216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10219.503733] Lustre: Skipped 1 previous similar message [10219.515163] Lustre: Skipped 1 previous similar message [10225.645399] Lustre: server umount lustre-MDT0000 complete [10229.729391] LustreError: 267571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.767841] LustreError: 267571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10230.186765] LustreError: 269528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786338 with bad export cookie 14541553741508157459 [10230.194797] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.205271] LustreError: 269528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10230.566917] Lustre: server umount lustre-MDT0001 complete [10252.770607] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10264.173902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.627179] LustreError: 272464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.691523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.702907] Lustre: Skipped 2 previous similar messages [10269.544486] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10278.867058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.235477] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10287.140634] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10294.204074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10300.633408] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10310.334348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10310.687836] Lustre: lustre-OST0001: new disk, initializing [10310.692967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10310.717190] Lustre: 274776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10312.207710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10312.248900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10312.415308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10320.396784] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10333.225361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10337.237199] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10342.993282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10343.113028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10351.073808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10351.087352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.104837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.110411] Lustre: Skipped 2 previous similar messages [10356.032652] Lustre: server umount lustre-MDT0000 complete [10358.764950] LustreError: 272459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.785549] LustreError: 272459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10359.797165] LustreError: 272446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786468 with bad export cookie 14541553741508158341 [10359.806956] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10359.816587] LustreError: 272446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10360.202230] Lustre: server umount lustre-MDT0001 complete [10374.896691] Lustre: server umount lustre-OST0000 complete [10389.307860] Lustre: server umount lustre-OST0001 complete [10407.127852] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10416.637736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.179188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10417.187188] Lustre: Skipped 3 previous similar messages [10421.461371] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10429.399450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.020276] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10438.616151] Lustre: 278970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10446.980447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10454.796339] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10461.685821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10463.227634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10468.875226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10470.797497] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10479.005193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10482.908854] Lustre: 280847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10488.627782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10488.657541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10497.729781] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:56:44 (1789786604) [10514.915908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.930502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10514.931826] Lustre: Skipped 4 previous similar messages [10514.948249] Lustre: Skipped 4 previous similar messages [10518.983402] Lustre: server umount lustre-MDT0000 complete [10520.039506] LustreError: 277827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.061915] LustreError: 277827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10522.962776] LustreError: 277813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786631 with bad export cookie 14541553741508165117 [10522.968548] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10522.979193] LustreError: 277813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10529.507625] Lustre: server umount lustre-MDT0001 complete [10548.194363] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10548.356543] Lustre: server umount lustre-OST0000 complete [10552.424899] Lustre: server umount lustre-OST0001 complete [10559.417705] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [10569.854888] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10619.933594] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10630.162274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.482349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10630.550932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10630.662732] Lustre: lustre-MDT0000: new disk, initializing [10630.755991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10635.424761] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10646.132620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10646.242429] Lustre: 285091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10646.556700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10651.328206] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10662.319797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10662.652385] Lustre: 286724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10664.915914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10670.378285] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10681.457026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10681.581759] Lustre: 287594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10681.734978] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10681.748757] Lustre: Skipped 6 previous similar messages [10683.523625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10683.528339] Lustre: Skipped 1 previous similar message [10683.591536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10688.546954] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10698.979893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10702.693401] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10702.706152] Lustre: Skipped 1 previous similar message [10717.200219] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:00:24 (1789786824) [10719.718661] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.726284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.737929] Lustre: Skipped 5 previous similar messages [10719.746683] Lustre: Skipped 7 previous similar messages [10721.418939] Lustre: server umount lustre-MDT0000 complete [10725.069186] LustreError: 287593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786833 with bad export cookie 14541553741508172537 [10725.076508] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10725.085221] LustreError: 287593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10725.412922] Lustre: server umount lustre-MDT0001 complete [10739.342123] Lustre: server umount lustre-OST0000 complete [10753.129351] Lustre: server umount lustre-OST0001 complete [10761.574164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.369930] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10773.384627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.715469] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10784.207606] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10790.425105] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10796.829455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10799.271046] LustreError: 291830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.295807] LustreError: 291830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10802.613786] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10809.442851] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10825.633185] loop1: detected capacity change from 0 to 204800000 [10825.833212] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10825.846971] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10825.877209] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10830.801677] loop1: detected capacity change from 0 to 204800000 [10836.096808] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.202214] LustreError: 289841:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs8976: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10836.212478] LustreError: 289841:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs8976 [10836.217519] LustreError: 289841:0:(mgs_handler.c:517:mgs_target_reg()) fs8976-MDT0001: Error checking target index: rc = -13 [10836.226695] LustreError: fs8976-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10836.235613] LustreError: 293569:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10836.241983] LustreError: 293569:0:(tgt_mount.c:2132:server_put_super()) no obd fs8976-MDT0001 [10836.245807] LustreError: 293569:0:(tgt_mount.c:116:server_deregister_mount()) fs8976-MDT0001 not registered [10836.280599] Lustre: server umount fs8976-MDT0001 complete [10836.284272] LustreError: 293569:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10844.781958] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.968617] Lustre: Setting parameter fs8976-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs8976-MDT0001 [10845.032790] Lustre: srv-fs8976-MDT0001: No data found on store. Initialize space. [10845.037527] Lustre: Skipped 4 previous similar messages [10845.107413] Lustre: fs8976-MDT0001: new disk, initializing [10845.114787] Lustre: Skipped 3 previous similar messages [10845.248781] Lustre: srv-fs8976-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10845.256810] Lustre: 294035:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8976-MDT0001-mdtlov: can't create new llog: rc = -115 [10845.263255] LustreError: 294035:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8976-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10849.400266] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10856.584726] Lustre: server umount fs8976-MDT0001 complete [10870.621568] loop1: detected capacity change from 0 to 204800000 [10876.955449] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.301942] Lustre: 295602:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8976-MDT0001-mdtlov: can't create new llog: rc = -115 [10877.313647] LustreError: 295602:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8976-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10881.848802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10883.568651] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10886.783918] Lustre: server umount fs8976-MDT0001 complete [10888.690870] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10896.864402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10896.873471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.884777] Lustre: Skipped 2 previous similar messages [10896.891858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.899656] Lustre: Skipped 3 previous similar messages [10902.415166] Lustre: server umount lustre-MDT0000 complete [10910.991065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10911.099386] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10915.586207] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10916.858954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10923.778770] loop1: detected capacity change from 0 to 204800000 [10929.054978] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.332597] Lustre: 298050:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8976-MDT0001-mdtlov: can't create new llog: rc = -115 [10929.343196] LustreError: 298050:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8976-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10932.858636] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10937.489592] Lustre: server umount fs8976-MDT0001 complete [10957.799433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10963.179839] Lustre: server umount lustre-OST0000 complete [10981.343126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10981.600466] Lustre: server umount lustre-MDT0000 complete [10985.054983] LustreError: 289834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787093 with bad export cookie 14541553741508175218 [10985.055943] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10985.063287] LustreError: 289834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10985.381395] Lustre: server umount lustre-MDT0001 complete [10995.329775] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10997.912707] Key type lgssc unregistered [10998.145653] LNet: 300393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10998.154672] LNetError: 300393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10998.173905] LNet: Removed LNI 192.168.203.142@tcp [10998.903655] Key type .llcrypt unregistered [10998.910117] Key type ._llcrypt unregistered [11016.084326] Key type ._llcrypt registered [11016.086567] Key type .llcrypt registered [11016.205468] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11017.151113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11017.322510] alg: No test for adler32 (adler32-zlib) [11018.420154] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11018.614986] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [11020.327159] Key type lgssc registered [11021.513632] Lustre: Echo OBD driver; http://www.lustre.org/ [11029.596522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11029.616247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.135701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11034.835532] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11041.251588] LustreError: 301661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.278885] LustreError: 301661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11042.737778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.265878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11048.152020] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11054.650907] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11060.530084] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11066.276386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11066.543678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11067.572520] LustreError: 303490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.838339] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11073.003084] LustreError: 303491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.080761] LustreError: 303490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.112645] LustreError: 303490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11080.022545] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11081.198714] LustreError: 303492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.213362] LustreError: 303492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11091.434399] LustreError: 303562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11091.445173] LustreError: 303562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11097.746630] loop1: detected capacity change from 0 to 204800000 [11097.864214] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11097.872671] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11097.882780] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11102.754704] loop1: detected capacity change from 0 to 204800000 [11107.523392] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.676395] Lustre: fs14532-OST0000: new disk, initializing [11107.685071] Lustre: srv-fs14532-OST0000: No data found on store. Initialize space. [11107.695385] Lustre: 305395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11107.856274] Lustre: fs14532-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11108.573446] LustreError: 303490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.592244] LustreError: 303490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11114.079854] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11123.098139] Lustre: server umount fs14532-OST0000 complete [11135.153089] loop1: detected capacity change from 0 to 204800000 [11140.787029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.134859] Lustre: fs14532-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11142.636561] LustreError: 303491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11142.665043] LustreError: 303491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11147.485682] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11152.890408] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11156.063175] loop2: detected capacity change from 0 to 204800000 [11156.310065] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11156.320594] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11156.329419] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11158.000360] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11161.676576] loop2: detected capacity change from 0 to 204800000 [11166.882764] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11166.979468] LustreError: fs14532-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11166.986073] LustreError: fs14532-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11166.992357] LustreError: 308436:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11166.997753] LustreError: 308436:0:(tgt_mount.c:2132:server_put_super()) no obd fs14532-OST0001 [11167.002581] LustreError: 308436:0:(tgt_mount.c:116:server_deregister_mount()) fs14532-OST0001 not registered [11167.033320] Lustre: server umount fs14532-OST0001 complete [11167.035478] LustreError: 308436:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11176.765760] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.931722] Lustre: fs14532-OST0001: new disk, initializing [11176.941768] Lustre: srv-fs14532-OST0001: No data found on store. Initialize space. [11176.954830] Lustre: 308988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11177.023329] Lustre: fs14532-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11181.655252] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11189.007355] Lustre: server umount fs14532-OST0000 complete [11198.916481] Lustre: server umount fs14532-OST0001 complete [11209.199714] LustreError: 304001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.222029] LustreError: 304001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11211.999536] loop1: detected capacity change from 0 to 204800000 [11218.926521] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11219.326746] Lustre: fs14532-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11225.167427] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11233.020373] loop2: detected capacity change from 0 to 204800000 [11238.536129] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11244.622662] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11253.216313] Lustre: server umount fs14532-OST0000 complete [11261.604402] Lustre: server umount fs14532-OST0001 complete [11275.749623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.768303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11275.769189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11275.788944] Lustre: Skipped 1 previous similar message [11280.884463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11280.890997] Lustre: Skipped 1 previous similar message [11281.338852] Lustre: server umount lustre-OST0000 complete [11285.987393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11285.992336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11286.040050] Lustre: Skipped 2 previous similar messages [11286.079478] Lustre: Skipped 1 previous similar message [11291.106581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11291.114761] Lustre: Skipped 1 previous similar message [11296.224759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11296.235490] Lustre: Skipped 1 previous similar message [11300.319145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11300.525362] Lustre: server umount lustre-MDT0000 complete [11304.748982] LustreError: 306141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787413 with bad export cookie 3909967870032087199 [11304.749617] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11304.771101] LustreError: 306141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11305.317139] Lustre: server umount lustre-MDT0001 complete [11317.372511] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [11320.648533] Key type lgssc unregistered [11320.947570] LNet: 314912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11320.955733] LNetError: 314912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11320.994391] LNet: Removed LNI 192.168.203.142@tcp [11321.952533] Key type .llcrypt unregistered [11321.959210] Key type ._llcrypt unregistered [11352.600305] Key type ._llcrypt registered [11352.603351] Key type .llcrypt registered [11352.713983] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [11365.307488] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11366.406676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11366.470173] alg: No test for adler32 (adler32-zlib) [11367.641313] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11367.960493] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [11369.623220] Key type lgssc registered [11370.808221] Lustre: Echo OBD driver; http://www.lustre.org/ [11406.475693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11406.498154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.868868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11407.905560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11408.020834] Lustre: lustre-MDT0000: new disk, initializing [11408.181870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11408.205209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11412.762842] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11423.797268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.932863] Lustre: 318750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11424.022503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11424.026788] Lustre: Skipped 1 previous similar message [11424.133101] Lustre: lustre-MDT0001: new disk, initializing [11424.272641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11424.321561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11424.344635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11429.745829] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11440.069138] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11446.176601] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11452.431850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11452.781285] Lustre: lustre-OST0000: new disk, initializing [11452.784853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11452.791597] Lustre: 320718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11452.904857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11459.155433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11459.186584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11459.365538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11460.452812] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11470.367481] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11472.341542] Lustre: Mounted lustre-client - version 2.17.57_111_g8cb7ca1 [11473.179272] Lustre: Unmounted lustre-client [11482.594970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11482.604642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11482.618274] Lustre: Skipped 1 previous similar message [11482.650283] Lustre: Skipped 1 previous similar message [11487.722327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11487.725559] Lustre: Skipped 1 previous similar message [11488.046887] Lustre: server umount lustre-OST0000 complete [11492.835484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11492.865105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11492.875050] Lustre: Skipped 1 previous similar message [11497.575308] Lustre: server umount lustre-MDT0000 complete [11501.229133] LustreError: 318740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787609 with bad export cookie 17378116329498706531 [11501.238522] LustreError: 318740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11501.240164] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11501.593163] Lustre: server umount lustre-MDT0001 complete [11511.073681] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [11513.496606] Key type lgssc unregistered [11524.127585] LNet: 322878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11524.144498] LNetError: 322878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11601.006185] LNet: Removed LNI 192.168.203.142@tcp [11601.869124] Key type .llcrypt unregistered [11601.872060] Key type ._llcrypt unregistered [11620.192982] Key type ._llcrypt registered [11620.203546] Key type .llcrypt registered [11620.360613] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [11637.522868] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11638.996541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11639.048739] alg: No test for adler32 (adler32-zlib) [11640.164566] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11640.382397] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [11642.041921] Key type lgssc registered [11643.071648] Lustre: Echo OBD driver; http://www.lustre.org/ [11676.265395] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 23:16:22 (1789787782) [11684.022670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11684.056781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.514682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11685.556785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11685.737705] Lustre: lustre-MDT0000: new disk, initializing [11685.912662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11685.943669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11691.708912] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11704.786277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11705.038852] Lustre: 326100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11705.133738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11705.146711] Lustre: Skipped 1 previous similar message [11705.292195] Lustre: lustre-MDT0001: new disk, initializing [11705.440105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11705.468278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11705.473829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11709.840183] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11722.312534] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11730.065718] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11737.740530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11738.165885] Lustre: lustre-OST0000: new disk, initializing [11738.173338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11738.194332] Lustre: 328072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11738.330449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11744.894896] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11747.925441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11747.937441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11748.107028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11754.786945] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11773.921680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11773.922435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11773.942940] LustreError: Skipped 1 previous similar message [11773.958667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11775.279457] Lustre: server umount lustre-OST0000 complete [11782.112051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11782.122804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11782.143255] Lustre: Skipped 1 previous similar message [11782.155852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11782.161613] Lustre: Skipped 1 previous similar message [11784.161310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11784.163325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.535956] Lustre: server umount lustre-MDT0000 complete [11789.389022] LustreError: 326897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787897 with bad export cookie 16255003723121696982 [11789.400291] LustreError: 326897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11789.402827] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11789.834995] Lustre: server umount lustre-MDT0001 complete [11802.068823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [11805.159025] Key type lgssc unregistered [11805.548441] LNet: 329976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11805.560623] LNetError: 329976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11805.600397] LNet: Removed LNI 192.168.203.142@tcp [11806.745140] Key type .llcrypt unregistered [11806.751598] Key type ._llcrypt unregistered [11828.634688] Key type ._llcrypt registered [11828.636911] Key type .llcrypt registered [11828.757704] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11830.242296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11830.299125] alg: No test for adler32 (adler32-zlib) [11831.445816] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11831.710329] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [11833.408666] Key type lgssc registered [11834.567029] Lustre: Echo OBD driver; http://www.lustre.org/ [11842.735203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11842.758818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.304958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11849.074970] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11853.802477] LustreError: 331247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.815473] LustreError: 331247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11858.095101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.384676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11863.587270] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11871.707920] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11878.283948] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11885.826356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11886.230886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11894.059234] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11903.415620] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11914.448895] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11916.784431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11923.428254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.436186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11923.452164] Lustre: Skipped 1 previous similar message [11923.469650] Lustre: Skipped 2 previous similar messages [11925.769431] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [11927.944795] Lustre: server umount lustre-MDT0000 complete [11928.549047] LustreError: 331242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.582553] LustreError: 331242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11930.865204] LustreError: 331244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.869621] LustreError: 331227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788041 with bad export cookie 6649771435172890570 [11932.880838] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11932.885455] LustreError: 331227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11933.665502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.667649] LustreError: 331243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.689912] Lustre: Skipped 1 previous similar message [11933.692704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11936.005260] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [11937.700898] LustreError: 331243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.723097] LustreError: 331243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11939.242679] Lustre: server umount lustre-MDT0001 complete [11943.673588] Lustre: server umount lustre-OST0000 complete [11953.497181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.607510] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11960.209298] Lustre: 335428:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.142@tcp [11962.304758] Lustre: server umount MGS complete [11970.097139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.517366] LustreError: 336000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.614404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11975.251124] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11983.731485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11984.052598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11988.571848] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [11996.887746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11997.295430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12003.667445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12011.224975] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12012.751979] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12018.298502] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12019.893457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12024.288990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12024.305049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.323923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12024.328977] Lustre: Skipped 1 previous similar message [12026.341614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12030.503475] Lustre: server umount lustre-OST0000 complete [12035.553034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12035.566421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12035.591173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.596690] Lustre: Skipped 2 previous similar messages [12040.973398] Lustre: server umount lustre-MDT0000 complete [12041.697968] LustreError: 336001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.725268] LustreError: 336001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12044.921397] LustreError: 335981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788153 with bad export cookie 6649771435172891403 [12044.929345] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12044.937717] LustreError: 335981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12045.446046] Lustre: server umount lustre-MDT0001 complete [12058.481236] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12061.407277] Key type lgssc unregistered [12061.712368] LNet: 339435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12061.719778] LNetError: 339435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12061.739334] LNet: Removed LNI 192.168.203.142@tcp [12062.494731] Key type .llcrypt unregistered [12062.496367] Key type ._llcrypt unregistered [12085.249171] Key type ._llcrypt registered [12085.251731] Key type .llcrypt registered [12085.399774] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [12103.880702] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [12105.058231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12105.174160] alg: No test for adler32 (adler32-zlib) [12106.434227] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [12106.703251] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [12108.360868] Key type lgssc registered [12109.644417] Lustre: Echo OBD driver; http://www.lustre.org/ [12147.828461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12147.859550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.131355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12149.261837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12149.324264] Lustre: lustre-MDT0000: new disk, initializing [12149.440092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12149.470397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12154.235871] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12165.863816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12166.025958] Lustre: 342595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12166.084331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12166.092997] Lustre: Skipped 1 previous similar message [12166.241576] Lustre: lustre-MDT0001: new disk, initializing [12166.403318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12166.493385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12166.513136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12171.430859] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12184.084483] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12192.222742] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12199.958356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12200.233389] Lustre: lustre-OST0000: new disk, initializing [12200.240161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12200.251101] Lustre: 344567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12200.366841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12206.162692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12206.182558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12206.353437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12206.839274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12216.999922] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12222.507638] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12222.758695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12228.752866] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12229.034635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12232.170335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12232.175982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.202978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12232.211501] Lustre: Skipped 1 previous similar message [12232.234904] Lustre: Skipped 1 previous similar message [12237.314750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12238.273691] Lustre: server umount lustre-OST0000 complete [12243.424591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12243.433069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12243.463134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12243.471387] Lustre: Skipped 1 previous similar message [12247.522590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12247.526514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12249.563133] Lustre: server umount lustre-MDT0000 complete [12253.456684] LustreError: 342589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788362 with bad export cookie 1423475536657820249 [12253.458677] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12253.470818] LustreError: 342589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12253.764469] Lustre: server umount lustre-MDT0001 complete [12263.827268] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:26:10 (1789788370) [12280.953927] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [12291.535448] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [12328.695619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.911575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12328.936603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12328.995500] Lustre: lustre-MDT0000: new disk, initializing [12329.048965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12329.063991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12332.733620] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12342.579974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.654321] Lustre: 349330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12342.707348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12342.713056] Lustre: Skipped 1 previous similar message [12342.793549] Lustre: lustre-MDT0001: new disk, initializing [12342.890284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.921544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12342.937610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12348.352407] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12362.414764] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12371.640357] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12378.959358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12379.169948] Lustre: lustre-OST0000: new disk, initializing [12379.173693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12379.178611] Lustre: 351298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12379.254632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12386.352489] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [12387.959093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12387.970072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12388.135906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12396.913973] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12434.408845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.418632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12434.421036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12434.421044] Lustre: Skipped 1 previous similar message [12434.431236] Lustre: Skipped 1 previous similar message [12434.463966] LustreError: Skipped 1 previous similar message [12444.127195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12444.421184] Lustre: server umount lustre-OST0000 complete [12449.763747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12449.775048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12449.784836] Lustre: Skipped 1 previous similar message [12449.808460] Lustre: Skipped 4 previous similar messages [12455.377472] Lustre: server umount lustre-MDT0000 complete [12459.297933] LustreError: 349322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788567 with bad export cookie 1423475536657821131 [12459.303704] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12459.315616] LustreError: 349322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12459.689200] Lustre: server umount lustre-MDT0001 complete [12472.346650] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12475.408843] Key type lgssc unregistered [12475.679778] LNet: 353474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12475.695257] LNetError: 353474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12475.710908] LNet: Removed LNI 192.168.203.142@tcp [12476.623889] Key type .llcrypt unregistered [12476.628034] Key type ._llcrypt unregistered [12572.415605] Key type ._llcrypt registered [12572.417138] Key type .llcrypt registered [12572.514345] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12572.737299] Key type .llcrypt unregistered [12572.738983] Key type ._llcrypt unregistered [12647.446596] Key type ._llcrypt registered [12647.450923] Key type .llcrypt registered [12647.600461] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12647.908539] Key type .llcrypt unregistered [12647.916548] Key type ._llcrypt unregistered [12674.178572] Key type ._llcrypt registered [12674.182540] Key type .llcrypt registered [12674.280727] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12674.502319] Key type .llcrypt unregistered [12674.505116] Key type ._llcrypt unregistered [12706.610407] Key type ._llcrypt registered [12706.615443] Key type .llcrypt registered [12706.740299] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12707.049137] Key type .llcrypt unregistered [12707.054246] Key type ._llcrypt unregistered [12785.445517] Key type ._llcrypt registered [12785.447512] Key type .llcrypt registered [12785.567739] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12785.907958] Key type .llcrypt unregistered [12785.912650] Key type ._llcrypt unregistered [12812.383424] Key type ._llcrypt registered [12812.386300] Key type .llcrypt registered [12812.519547] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12812.923399] Key type .llcrypt unregistered [12812.931586] Key type ._llcrypt unregistered [12843.591343] Key type ._llcrypt registered [12843.594700] Key type .llcrypt registered [12843.691579] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12843.928444] Key type .llcrypt unregistered [12843.930206] Key type ._llcrypt unregistered [12870.417366] Key type ._llcrypt registered [12870.420280] Key type .llcrypt registered [12870.506961] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12870.777288] Key type .llcrypt unregistered [12870.780427] Key type ._llcrypt unregistered [12905.832355] Key type ._llcrypt registered [12905.834256] Key type .llcrypt registered [12906.000636] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12906.514579] Key type .llcrypt unregistered [12906.517111] Key type ._llcrypt unregistered [12939.166335] Key type ._llcrypt registered [12939.167849] Key type .llcrypt registered [12939.313126] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12939.738338] Key type .llcrypt unregistered [12939.740668] Key type ._llcrypt unregistered [12999.320970] Key type ._llcrypt registered [12999.322438] Key type .llcrypt registered [12999.399176] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [12999.688316] Key type .llcrypt unregistered [12999.691676] Key type ._llcrypt unregistered [13031.001884] Key type ._llcrypt registered [13031.003524] Key type .llcrypt registered [13031.092172] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [13031.470563] Key type .llcrypt unregistered [13031.484772] Key type ._llcrypt unregistered [13065.520075] Key type ._llcrypt registered [13065.521736] Key type .llcrypt registered [13065.618501] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [13065.908619] Key type .llcrypt unregistered [13065.911533] Key type ._llcrypt unregistered [13118.534260] Key type ._llcrypt registered [13118.536347] Key type .llcrypt registered [13118.723736] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [13136.014836] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [13137.296039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13137.433064] alg: No test for adler32 (adler32-zlib) [13138.520642] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [13138.724174] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [13140.407156] Key type lgssc registered [13141.335988] Lustre: Echo OBD driver; http://www.lustre.org/ [13177.021854] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13177.063618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.437529] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13178.486412] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13178.616603] Lustre: tst250fs-MDT0000: new disk, initializing [13178.754334] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13178.770904] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13184.323670] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [13195.383705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13195.560785] Lustre: 373659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13195.662824] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13195.668817] Lustre: Skipped 1 previous similar message [13195.760891] Lustre: tst250fs-MDT0001: new disk, initializing [13195.887154] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13195.930414] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13195.943779] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13199.871699] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [13205.420628] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13214.020054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13214.399242] Lustre: tst250fs-OST0000: new disk, initializing [13214.405489] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13214.414601] Lustre: 375290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13214.525942] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13216.248959] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13216.258915] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13216.325278] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13221.707594] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [13231.145150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13231.255555] Lustre: tst250fs-OST0001: new disk, initializing [13231.258771] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13231.263185] Lustre: 376163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13231.371374] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13237.018256] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [13239.911865] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13239.920121] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13239.957344] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13242.479662] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13244.857882] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13244.866752] Lustre: Skipped 2 previous similar messages [13260.777045] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13260.778901] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13260.809419] Lustre: Skipped 3 previous similar messages [13260.811905] Lustre: Skipped 2 previous similar messages [13265.890734] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13265.899091] Lustre: Skipped 4 previous similar messages [13266.751514] Lustre: server umount tst250fs-MDT0000 complete [13269.642472] LustreError: 373649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789378 with bad export cookie 4573457158185178978 [13269.646950] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13269.656233] LustreError: 373649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13269.941895] Lustre: server umount tst250fs-MDT0001 complete [13283.443126] Lustre: server umount tst250fs-OST0000 complete [13297.029349] Lustre: server umount tst250fs-OST0001 complete [13312.810750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [13321.019195] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [13350.094754] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:44:16 (1789789456) [13351.572991] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13366.053485] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [13375.559902] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [13405.813403] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13115 sec ========= 23:45:12 (1789789512) [13407.388897] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:45:14 (1789789514) === [13411.169281] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:45:17 (1789789517) === [13449.081893] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [13451.498747] Key type lgssc unregistered [13451.781777] LNet: 385301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13451.791354] LNetError: 385301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13451.809957] LNet: Removed LNI 192.168.203.142@tcp [13452.666248] Key type .llcrypt unregistered [13452.668090] Key type ._llcrypt unregistered