[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 405277950 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002331] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006869] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012061] *** VALIDATE selinux *** [ 0.020655] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024568] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028085] *** VALIDATE tmpfs *** [ 0.029457] *** VALIDATE proc *** [ 0.030150] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.033139] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.039897] debug: unmapping init [mem 0xffffffffa7859000-0xffffffffa7860fff] [ 0.041819] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042544] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043017] ... version: 2 [ 0.044006] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.045988] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048007] ... event mask: 000000070000000f [ 0.049298] rcu: Hierarchical SRCU implementation. [ 0.051238] smp: Bringing up secondary CPUs ... [ 0.052441] x86: Booting SMP configuration: [ 0.053019] .... node #0, CPUs: #1 #2 #3 [ 0.056127] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.059026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.189532] node 0 deferred pages initialised in 130ms [ 0.192621] devtmpfs: initialized [ 0.193166] x86/mm: Memory block size: 128MB [ 0.195715] gcov: version magic: 0x41383552 [ 0.198170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.200091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.202372] pinctrl core: initialized pinctrl subsystem [ 0.204163] [ 0.204804] ************************************************************* [ 0.206013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208012] ** ** [ 0.210011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.211008] ** ** [ 0.213013] ** This means that this kernel is built to expose internal ** [ 0.215012] ** IOMMU data structures, which may compromise security on ** [ 0.217011] ** your system. ** [ 0.219012] ** ** [ 0.221013] ** If you see this message and you are not debugging the ** [ 0.223015] ** kernel, report this immediately to your vendor! ** [ 0.225011] ** ** [ 0.227010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229018] ************************************************************* [ 0.231702] NET: Registered protocol family 16 [ 0.233543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.236048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.238078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241415] cpuidle: using governor menu [ 0.243257] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245440] PCI: Using configuration type 1 for base access [ 0.247107] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.254098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.256022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.259116] cryptd: max_cpu_qlen set to 1000 [ 0.261236] ACPI: Added _OSI(Module Device) [ 0.263011] ACPI: Added _OSI(Processor Device) [ 0.264021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269463] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275383] ACPI: Interpreter enabled [ 0.276038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.277010] ACPI: Using IOAPIC for interrupt routing [ 0.279091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.290391] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.292021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.301043] acpiphp: Slot [2] registered [ 0.302099] acpiphp: Slot [5] registered [ 0.303096] acpiphp: Slot [6] registered [ 0.304144] acpiphp: Slot [7] registered [ 0.306085] acpiphp: Slot [8] registered [ 0.307117] acpiphp: Slot [9] registered [ 0.309069] acpiphp: Slot [10] registered [ 0.310109] acpiphp: Slot [3] registered [ 0.311047] acpiphp: Slot [4] registered [ 0.312067] acpiphp: Slot [11] registered [ 0.313053] acpiphp: Slot [12] registered [ 0.315070] acpiphp: Slot [13] registered [ 0.316105] acpiphp: Slot [14] registered [ 0.318139] acpiphp: Slot [15] registered [ 0.319105] acpiphp: Slot [16] registered [ 0.321097] acpiphp: Slot [17] registered [ 0.322093] acpiphp: Slot [18] registered [ 0.323082] acpiphp: Slot [19] registered [ 0.325100] acpiphp: Slot [20] registered [ 0.326087] acpiphp: Slot [21] registered [ 0.327072] acpiphp: Slot [22] registered [ 0.328063] acpiphp: Slot [23] registered [ 0.329071] acpiphp: Slot [24] registered [ 0.331064] acpiphp: Slot [25] registered [ 0.332084] acpiphp: Slot [26] registered [ 0.334113] acpiphp: Slot [27] registered [ 0.335095] acpiphp: Slot [28] registered [ 0.336062] acpiphp: Slot [29] registered [ 0.338070] acpiphp: Slot [30] registered [ 0.339081] acpiphp: Slot [31] registered [ 0.340049] PCI host bridge to bus 0000:00 [ 0.342018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344052] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357936] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361071] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373759] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383520] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386578] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390777] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411904] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453482] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495082] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.541118] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.547014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.553023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.569019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580130] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.585015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.604014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.613788] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.621987] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645712] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.650228] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652306] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655341] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.661123] iommu: Default domain type: Passthrough [ 0.662480] SCSI subsystem initialized [ 0.664169] ACPI: bus type USB registered [ 0.665098] usbcore: registered new interface driver usbfs [ 0.667075] usbcore: registered new interface driver hub [ 0.669081] usbcore: registered new device driver usb [ 0.670107] pps_core: LinuxPPS API ver. 1 registered [ 0.672007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675047] PTP clock support registered [ 0.676125] EDAC MC: Ver: 3.0.0 [ 0.677320] PCI: Using ACPI for IRQ routing [ 0.679869] NetLabel: Initializing [ 0.681011] NetLabel: domain hash size = 128 [ 0.682009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684081] NetLabel: unlabeled traffic allowed by default [ 0.686135] vgaarb: loaded [ 0.687288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.690013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694008] clocksource: Switched to clocksource kvm-clock [ 0.787706] VFS: Disk quotas dquot_6.6.0 [ 0.789279] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.791192] *** VALIDATE ramfs *** [ 0.792105] *** VALIDATE hugetlbfs *** [ 0.793507] pnp: PnP ACPI init [ 0.795725] pnp: PnP ACPI: found 6 devices [ 0.809278] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812268] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814052] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.815895] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.818545] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.820324] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.822918] NET: Registered protocol family 2 [ 0.825272] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.829749] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.833695] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.839107] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.842423] TCP: Hash tables configured (established 65536 bind 65536) [ 0.845219] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.847453] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849922] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853024] NET: Registered protocol family 1 [ 0.855677] RPC: Registered named UNIX socket transport module. [ 0.857982] RPC: Registered udp transport module. [ 0.859172] RPC: Registered tcp transport module. [ 0.860504] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.862489] NET: Registered protocol family 44 [ 0.863739] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865351] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.866931] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.869232] PCI: CLS 0 bytes, default 64 [ 0.871762] Unpacking initramfs... [ 2.196212] debug: unmapping init [mem 0xffff8c203cc54000-0xffff8c203ffbffff] [ 2.199448] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.200961] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.203737] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.654276] Initialise system trusted keyrings [ 2.655974] Key type blacklist registered [ 2.657641] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.665832] zbud: loaded [ 2.668619] *** VALIDATE nfs *** [ 2.669784] *** VALIDATE nfs4 *** [ 2.671037] pstore: using deflate compression [ 2.674060] Platform Keyring initialized [ 2.771802] NET: Registered protocol family 38 [ 2.773730] Key type asymmetric registered [ 2.774762] Asymmetric key parser 'x509' registered [ 2.775974] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.778268] io scheduler mq-deadline registered [ 2.779880] io scheduler kyber registered [ 2.780996] io scheduler bfq registered [ 2.782211] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.784967] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.787225] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.789747] ACPI: Power Button [PWRF] [ 2.793837] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.799221] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.809816] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.816552] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.832462] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.860467] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.889562] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.894767] Non-volatile memory driver v1.3 [ 2.896133] Linux agpgart interface v0.103 [ 2.929542] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.934473] vda: detected capacity change from 0 to 74326016 [ 2.948037] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.950349] vdb: detected capacity change from 0 to 1073741824 [ 2.962211] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.964961] vdc: detected capacity change from 0 to 2621440000 [ 2.976595] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.979105] vdd: detected capacity change from 0 to 2621440000 [ 2.990637] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.993288] vde: detected capacity change from 0 to 4294967296 [ 3.007783] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.010204] vdf: detected capacity change from 0 to 4294967296 [ 3.017131] libphy: Fixed MDIO Bus: probed [ 3.021859] usbcore: registered new interface driver usbserial_generic [ 3.024145] usbserial: USB Serial support registered for generic [ 3.026198] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.030050] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.031563] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.033599] mousedev: PS/2 mouse device common for all mice [ 3.036046] rtc_cmos 00:05: RTC can wake from S4 [ 3.038948] rtc_cmos 00:05: registered as rtc0 [ 3.039370] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.040796] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.046470] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.047106] intel_pstate: CPU model not supported [ 3.051971] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.056069] hid: raw HID events driver (C) Jiri Kosina [ 3.057977] usbcore: registered new interface driver usbhid [ 3.059943] usbhid: USB HID core driver [ 3.061320] drop_monitor: Initializing network drop monitor service [ 3.063617] Initializing XFRM netlink socket [ 3.065099] NET: Registered protocol family 10 [ 3.067835] Segment Routing with IPv6 [ 3.069046] NET: Registered protocol family 17 [ 3.070794] mpls_gso: MPLS GSO support [ 3.076547] RAS: Correctable Errors collector initialized. [ 3.079215] AVX version of gcm_enc/dec engaged. [ 3.080365] AES CTR mode by8 optimization enabled [ 3.137993] sched_clock: Marking stable (3137965777, 0)->(3903579204, -765613427) [ 3.141528] registered taskstats version 1 [ 3.143037] Loading compiled-in X.509 certificates [ 3.144857] zswap: loaded using pool lzo/zbud [ 3.162689] Key type big_key registered [ 3.173044] Key type encrypted registered [ 3.174658] ima: No TPM chip found, activating TPM-bypass! [ 3.176388] ima: Allocated hash algorithm: sha1 [ 3.177882] ima: No architecture policies found [ 3.179497] evm: Initialising EVM extended attributes: [ 3.181124] evm: security.selinux [ 3.182129] evm: security.ima [ 3.182834] evm: security.capability [ 3.183884] evm: HMAC attrs: 0x1 [ 3.186170] rtc_cmos 00:05: setting system clock to 2026-07-04 06:21:28 UTC (1783146088) [ 3.191570] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.194371] debug: unmapping init [mem 0xffffffffa7582000-0xffffffffa7858fff] [ 3.203083] Write protecting the kernel read-only data: 28672k [ 3.206317] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 3.208715] debug: unmapping init [mem 0xffffffffa6514000-0xffffffffa65fffff] [ 3.238740] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.248115] systemd[1]: Detected virtualization kvm. [ 3.250128] systemd[1]: Detected architecture x86-64. [ 3.252241] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.278821] systemd[1]: No hostname configured. [ 3.280711] systemd[1]: Set hostname to . [ 3.283024] random: systemd: uninitialized urandom read (16 bytes read) [ 3.285739] systemd[1]: Initializing machine ID from random generator. [ 3.328391] random: ln: uninitialized urandom read (6 bytes read) [ 3.391550] random: systemd: uninitialized urandom read (16 bytes read) [ 3.394381] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.398340] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.402021] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.972042] device-mapper: uevent: version 1.0.3 [ 3.973855] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.628355] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.639623] virtio_net virtio0 ens2: renamed from eth0 [ 4.658829] scsi host0: ata_piix [ 4.679270] scsi host1: ata_piix [ 4.680937] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.683199] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.874783] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.688391] random: crng init done [ 9.689668] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.911090] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.992351] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.224027] SELinux: Disabled at runtime. [ 11.280592] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.287963] systemd[1]: Detected virtualization kvm. [ 11.289625] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.704400] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.707688] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.711862] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.715482] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.718988] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.725979] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.734530] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ 11.756757] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.056742] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.323130] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.337929] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.464378] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.472440] EDAC sbridge: Ver: 1.1.2 [ 14.073344] Key type dns_resolver registered [ 14.354321] NFS: Registering the id_resolver key type [ 14.355873] Key type id_resolver registered [ 14.357500] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-server login: [ 35.509973] libcfs: loading out-of-tree module taints kernel. [ 35.526973] Key type ._llcrypt registered [ 35.528377] Key type .llcrypt registered [ 35.568336] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 43.086325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 43.579141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 43.585692] alg: No test for adler32 (adler32-zlib) [ 44.559065] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 44.837076] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 46.439155] Key type lgssc registered [ 47.068792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 53.218572] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.506703] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 70.658535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 70.665462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 71.763706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 71.774631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 71.813551] Lustre: lustre-MDT0000: new disk, initializing [ 71.836276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 71.842210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 73.128095] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 78.465139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.501349] Lustre: 6477:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 78.516128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.518803] Lustre: Skipped 1 previous similar message [ 78.561444] Lustre: lustre-MDT0001: new disk, initializing [ 78.585109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.596851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.601487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.962689] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.222896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 85.284596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 85.381110] Lustre: lustre-OST0000: new disk, initializing [ 85.383265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 85.385735] Lustre: 8415:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 85.403582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.338443] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 89.132956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 89.136267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 89.163389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 92.928314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.978585] Lustre: lustre-OST0001: new disk, initializing [ 92.980274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 92.982589] Lustre: 9488:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 93.002295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.929082] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.685047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 96.688899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 96.703937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 100.756441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 105.344329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 107.223451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing check_logdir /tmp/testlogs/ [ 108.658861] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing yml_node [ 110.023761] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 110.818810] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 111.633413] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 112.185056] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 4 02:23:16 EDT 2026 [ 117.787504] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 118.322282] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 130.015572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 130.019113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 130.024118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 132.575506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 132.576333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 132.581927] Lustre: Skipped 1 previous similar message [ 137.695774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.698433] Lustre: Skipped 6 previous similar messages [ 142.815847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 142.817837] Lustre: Skipped 3 previous similar messages [ 144.351140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 144.409292] Lustre: server umount lustre-MDT0000 complete [ 145.647706] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146230 with bad export cookie 6588309134534833384 [ 145.648888] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 145.652456] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 147.935738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.936037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.939097] Lustre: Skipped 2 previous similar messages [ 151.842331] Lustre: server umount lustre-MDT0001 complete [ 167.391136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 167.429445] Lustre: server umount lustre-OST0000 complete [ 174.797108] Lustre: server umount lustre-OST0001 complete [ 177.014361] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 180.173369] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 192.340129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.436892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 192.450576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.487916] Lustre: lustre-MDT0000: new disk, initializing [ 192.512422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.520721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.877419] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 198.117991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.160934] Lustre: 15031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.290489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.738783] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 203.543613] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 205.426666] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 207.727768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.841880] Lustre: lustre-OST0000: new disk, initializing [ 207.844422] Lustre: Skipped 1 previous similar message [ 207.847130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.850048] Lustre: Skipped 2 previous similar messages [ 207.853104] Lustre: 17000:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.887659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.890722] Lustre: Skipped 1 previous similar message [ 209.094592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.098246] Lustre: Skipped 1 previous similar message [ 209.100997] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.150734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.961352] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 213.759649] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 215.263308] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 215.348466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 216.781178] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 216.849069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 219.618216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.618365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 219.622911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 219.625086] LustreError: Skipped 1 previous similar message [ 219.627631] Lustre: Skipped 3 previous similar messages [ 223.571532] Lustre: server umount lustre-OST0000 complete [ 229.344073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 229.347314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.353871] Lustre: Skipped 1 previous similar message [ 231.008779] Lustre: server umount lustre-MDT0000 complete [ 232.390119] LustreError: 15024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146317 with bad export cookie 6588309134534834903 [ 232.393310] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 232.395058] LustreError: 15024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 232.527346] Lustre: server umount lustre-MDT0001 complete [ 234.489346] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:25:19 (1783146319) [ 240.936616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.101393] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 241.108544] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 241.128184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.506459] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 245.404300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.995787] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 249.522982] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 251.465888] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 252.267276] Lustre: Failing over lustre-MDT0000 [ 252.357141] Lustre: server umount lustre-MDT0000 complete [ 255.328817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.329420] LustreError: 19896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.332625] Lustre: Skipped 2 previous similar messages [ 260.575972] LustreError: 19871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.581944] LustreError: 19871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 265.556644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 265.600364] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.181184] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 269.792640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 269.795610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 269.801748] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 270.990689] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 274.912491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 274.912694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 274.913334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 274.913340] Lustre: Skipped 2 previous similar messages [ 274.918210] Lustre: Skipped 1 previous similar message [ 279.124303] Lustre: server umount lustre-MDT0000 complete [ 280.032097] LustreError: 19871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.037867] LustreError: 19871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 280.420370] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146365 with bad export cookie 6588309134534836317 [ 280.423407] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.425600] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.549767] Lustre: server umount lustre-MDT0001 complete [ 283.686799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.908026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.910496] Lustre: Skipped 2 previous similar messages [ 285.351779] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 288.205043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.738107] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 292.094658] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.956598] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 294.735875] Lustre: Failing over lustre-MDT0000 [ 294.828274] Lustre: server umount lustre-MDT0000 complete [ 297.825387] LustreError: 23201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.831385] LustreError: 23201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 307.752638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.789772] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.224374] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 312.993516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 312.996384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 312.999524] Lustre: Skipped 1 previous similar message [ 313.006247] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 313.015940] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 315.015222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.015847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.019503] Lustre: Skipped 3 previous similar messages [ 315.024105] Lustre: Skipped 2 previous similar messages [ 321.178361] Lustre: server umount lustre-MDT0000 complete [ 322.433357] LustreError: 23935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146407 with bad export cookie 6588309134534837255 [ 322.436883] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.438447] LustreError: 23935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.550601] Lustre: server umount lustre-MDT0001 complete [ 325.567228] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:26:50 (1783146410) [ 331.160198] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 334.506338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 347.096481] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 350.765581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.880667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 350.892732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 350.931093] Lustre: lustre-MDT0000: new disk, initializing [ 350.957979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 350.960154] Lustre: Skipped 2 previous similar messages [ 350.966239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 352.360442] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 356.724245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.770826] Lustre: 29566:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 356.896909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 358.337451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 362.021949] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.884948] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 366.182477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 366.314284] Lustre: 31532:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 367.744720] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 367.767150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 367.774076] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 368.302973] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 372.071059] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 374.413517] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 380.777960] Lustre: server umount lustre-OST0000 complete [ 382.431891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 382.434767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.439595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 382.442201] Lustre: Skipped 2 previous similar messages [ 392.701580] Lustre: server umount lustre-MDT0000 complete [ 393.696054] LustreError: 29577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.702283] LustreError: 29577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 393.877232] LustreError: 30360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146479 with bad export cookie 6588309134534837661 [ 393.880737] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.881034] LustreError: 30360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 404.504346] Lustre: server umount lustre-MDT0001 complete [ 408.110122] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 409.128719] Key type lgssc unregistered [ 409.257362] LNet: 33484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.260516] LNetError: 33484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.268275] LNet: Removed LNI 192.168.204.105@tcp [ 409.613118] Key type .llcrypt unregistered [ 409.614727] Key type ._llcrypt unregistered [ 418.590733] Key type ._llcrypt registered [ 418.592327] Key type .llcrypt registered [ 418.627175] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 419.020151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 419.028056] alg: No test for adler32 (adler32-zlib) [ 419.861991] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 419.940100] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 421.519107] Key type lgssc registered [ 421.834515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 424.197184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 424.202976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.340902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.544306] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 429.090686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.191250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.386886] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 432.605153] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.365508] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 436.470780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.581330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.356303] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 439.656635] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 440.543726] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.382538] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:29:00 (1783146540) [ 455.956309] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 456.503552] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 458.523344] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:29:03 (1783146543) [ 461.140483] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:29:05 (1783146545) [ 461.478583] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 464.132272] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 464.134738] Lustre: Skipped 2 previous similar messages [ 464.649270] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:29:09 (1783146549) [ 471.218624] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 471.221333] Lustre: Skipped 10 previous similar messages [ 472.078814] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:29:16 (1783146556) [ 472.962919] Lustre: *** cfs_fail_loc=131b, val=2*** [ 474.080261] Lustre: *** cfs_fail_loc=131b, val=2*** [ 476.779225] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:29:21 (1783146561) [ 480.892441] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 480.894825] Lustre: Skipped 7 previous similar messages [ 481.450201] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:29:26 (1783146566) [ 485.857117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.857241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.859710] LustreError: Skipped 1 previous similar message [ 485.864280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.867813] Lustre: Skipped 1 previous similar message [ 490.977115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.979645] Lustre: Skipped 1 previous similar message [ 491.213771] Lustre: server umount lustre-OST0000 complete [ 496.096753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 496.097046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.099733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.103428] Lustre: Skipped 2 previous similar messages [ 498.583983] Lustre: server umount lustre-MDT0000 complete [ 499.715485] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146585 with bad export cookie 13172148689432941376 [ 499.717851] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.719260] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 499.822633] Lustre: server umount lustre-MDT0001 complete [ 503.229908] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 504.155795] Key type lgssc unregistered [ 504.282267] LNet: 42719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.285046] LNetError: 42719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.293254] LNet: Removed LNI 192.168.204.105@tcp [ 504.568205] Key type .llcrypt unregistered [ 504.569331] Key type ._llcrypt unregistered [ 510.264078] Key type ._llcrypt registered [ 510.265621] Key type .llcrypt registered [ 510.303499] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 516.299065] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 516.592733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 516.629189] alg: No test for adler32 (adler32-zlib) [ 517.479686] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 517.561472] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 519.143141] Key type lgssc registered [ 519.497832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 530.537961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 530.544096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.626285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 531.637836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 531.671735] Lustre: lustre-MDT0000: new disk, initializing [ 531.712150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.718685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 533.075866] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 537.000549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.043443] Lustre: 45845:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 537.066584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 537.069277] Lustre: Skipped 1 previous similar message [ 537.104900] Lustre: lustre-MDT0001: new disk, initializing [ 537.134650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 537.145615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 537.149463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 538.384179] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 541.976955] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 543.604847] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 545.744820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.841340] Lustre: lustre-OST0000: new disk, initializing [ 545.842934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 545.845043] Lustre: 47811:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 545.869184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.740975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 546.744821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 546.771687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 547.656540] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 551.218467] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 552.675542] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 552.742539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 554.121093] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 554.197361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 557.026401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.026407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.027189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.029968] Lustre: Skipped 1 previous similar message [ 562.145594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.147467] Lustre: Skipped 2 previous similar messages [ 567.267196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.269097] Lustre: Skipped 1 previous similar message [ 569.311162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 569.370098] Lustre: server umount lustre-OST0000 complete [ 572.384581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.385183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.389174] Lustre: Skipped 1 previous similar message [ 576.726971] Lustre: server umount lustre-MDT0000 complete [ 577.934858] LustreError: 45836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146663 with bad export cookie 7505621802021977788 [ 577.937495] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.939370] LustreError: 45836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 578.046147] Lustre: server umount lustre-MDT0001 complete [ 581.254905] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:31:06 (1783146666) [ 583.699367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.859078] LustreError: 50268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.866242] LustreError: 50268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 583.887951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.343402] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 588.309937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.442274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 589.787520] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 592.193510] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.020621] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 596.141661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.005724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 600.178223] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 612.446340] LustreError: 52815:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 612.732121] Lustre: Modifying parameter general.timeout=1 in log params [ 613.793059] Lustre: Modifying parameter general.timeout=5 in log params [ 613.795128] Lustre: Skipped 3 previous similar messages [ 615.996983] Lustre: Modifying parameter general.timeout=13 in log params [ 615.998636] Lustre: Skipped 7 previous similar messages [ 620.222846] Lustre: Modifying parameter general.timeout=28 in log params [ 620.225147] Lustre: Skipped 14 previous similar messages [ 628.246235] Lustre: Modifying parameter general.timeout=57 in log params [ 628.248613] Lustre: Skipped 28 previous similar messages [ 644.540365] Lustre: Modifying parameter general.timeout=114 in log params [ 644.542526] Lustre: Skipped 56 previous similar messages [ 652.505731] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:32:17 (1783146737) [ 653.447410] Lustre: 59713:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 654.622843] Lustre: 59909:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 654.627836] Lustre: 59909:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 665.036393] Lustre: 61548:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 665.041831] Lustre: 61548:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 3 previous similar messages [ 665.049841] LustreError: 61548:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 666.364615] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:32:31 (1783146751) [ 670.179583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.180913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.183918] Lustre: Skipped 1 previous similar message [ 670.186260] Lustre: Skipped 1 previous similar message [ 676.070229] Lustre: server umount lustre-MDT0000 complete [ 677.242191] LustreError: 50250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146762 with bad export cookie 7505621802021978656 [ 677.244413] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.245942] LustreError: 50250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 677.360923] Lustre: server umount lustre-MDT0001 complete [ 687.832147] Lustre: server umount lustre-OST0000 complete [ 696.170218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.265479] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 696.275749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 696.279504] Lustre: Skipped 38 previous similar messages [ 696.360245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.362012] Lustre: Skipped 1 previous similar message [ 697.619221] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 700.323678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.355932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 700.364458] Lustre: 64070:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 701.862020] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 704.290798] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 706.033420] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 708.121625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.184082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 709.997251] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 712.195135] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 712.697726] Lustre: 66634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 716.258230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 716.259600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.261704] LustreError: Skipped 2 previous similar messages [ 716.265791] Lustre: Skipped 1 previous similar message [ 716.266534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 716.271473] Lustre: Skipped 6 previous similar messages [ 720.648846] Lustre: server umount lustre-OST0000 complete [ 725.983720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.986112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.991541] Lustre: Skipped 1 previous similar message [ 735.712229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.714200] Lustre: Skipped 6 previous similar messages [ 736.223085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 736.279923] Lustre: server umount lustre-MDT0000 complete [ 737.404055] LustreError: 66636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146822 with bad export cookie 7505621802021980945 [ 737.409642] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.504184] Lustre: server umount lustre-MDT0001 complete [ 740.955356] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 741.846585] Key type lgssc unregistered [ 741.965303] LNet: 67839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.968237] LNetError: 67839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.978268] LNet: Removed LNI 192.168.204.105@tcp [ 742.250496] Key type .llcrypt unregistered [ 742.251567] Key type ._llcrypt unregistered [ 751.200209] Key type ._llcrypt registered [ 751.201677] Key type .llcrypt registered [ 751.238798] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 751.599294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.631060] alg: No test for adler32 (adler32-zlib) [ 752.495403] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 752.593965] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 754.183103] Key type lgssc registered [ 754.548357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.271993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.277952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.425271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.857316] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 762.615037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.798374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 764.119347] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 766.364724] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.046469] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 770.192731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 770.362561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 772.116130] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 774.254968] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 787.482039] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 793.568159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 793.572385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.577072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.082712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.082792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.088250] Lustre: Skipped 2 previous similar messages [ 796.121832] Lustre: server umount lustre-MDT0000 complete [ 797.332703] LustreError: 69208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146882 with bad export cookie 15116370765631958322 [ 797.335501] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 797.337263] LustreError: 69208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 797.443501] Lustre: server umount lustre-MDT0001 complete [ 808.991837] Lustre: server umount lustre-OST0000 complete [ 817.779461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.871291] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 817.881278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 817.968469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.250767] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 822.027437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.055766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 822.065299] Lustre: 74115:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 822.227093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.546091] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 825.900681] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 827.581689] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.722755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.788673] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 831.841194] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 834.175489] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 839.872340] Lustre: 76686:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 840.472208] Lustre: Modifying parameter general.jobid_var=disable in log params [ 845.283468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.284227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 845.289345] Lustre: Skipped 1 previous similar message [ 845.294660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 845.297684] Lustre: Skipped 1 previous similar message [ 847.760662] Lustre: server umount lustre-OST0000 complete [ 850.401616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.402562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.406810] Lustre: Skipped 1 previous similar message [ 855.191696] Lustre: server umount lustre-MDT0000 complete [ 855.519972] LustreError: 74125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.526891] LustreError: 74125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 856.419958] LustreError: 76693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146941 with bad export cookie 15116370765631959141 [ 856.423499] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 856.424707] LustreError: 76693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 856.537649] Lustre: server umount lustre-MDT0001 complete [ 860.039451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 861.022196] Key type lgssc unregistered [ 861.145295] LNet: 77894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.147472] LNetError: 77894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.155298] LNet: Removed LNI 192.168.204.105@tcp [ 861.450440] Key type .llcrypt unregistered [ 861.451484] Key type ._llcrypt unregistered [ 870.538807] Key type ._llcrypt registered [ 870.539802] Key type .llcrypt registered [ 870.574439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 871.013832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.035120] alg: No test for adler32 (adler32-zlib) [ 871.895764] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 871.994098] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 873.583148] Key type lgssc registered [ 873.955159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.557345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.563404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.711167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.043078] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 881.741291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.946277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 883.295911] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 885.673101] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 887.560816] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 889.805161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 889.994276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.914338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 894.202193] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 906.658453] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 907.247948] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 907.250220] Lustre: Skipped 1 previous similar message [ 908.437043] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 908.439899] Lustre: Skipped 3 previous similar messages [ 910.541545] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 910.543547] Lustre: Skipped 6 previous similar messages [ 914.782508] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 914.785116] Lustre: Skipped 13 previous similar messages [ 922.836955] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 922.839931] Lustre: Skipped 26 previous similar messages [ 939.025284] Lustre: 87376:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 939.353412] Lustre: Disabling parameter general.jobid_var= in log params [ 939.356100] Lustre: Skipped 45 previous similar messages [ 939.873587] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:37:04 (1783147024) [ 940.406495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 940.954692] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:37:05 (1783147025) [ 943.584139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.587029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.591475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.610875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.613887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.619437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.729880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.731597] Lustre: Skipped 2 previous similar messages [ 954.850323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.852630] Lustre: Skipped 1 previous similar message [ 956.383089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 956.442626] Lustre: server umount lustre-OST0000 complete [ 958.944131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 958.946649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.950732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.871573] Lustre: server umount lustre-MDT0000 complete [ 965.088369] LustreError: 79291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.093344] LustreError: 79291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 965.162648] LustreError: 79269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147050 with bad export cookie 1688098101249626407 [ 965.167910] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.299368] Lustre: server umount lustre-MDT0001 complete [ 968.996532] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 970.114834] Key type lgssc unregistered [ 970.248436] LNet: 88625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.252265] LNetError: 88625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.262339] LNet: Removed LNI 192.168.204.105@tcp [ 970.597400] Key type .llcrypt unregistered [ 970.598569] Key type ._llcrypt unregistered [ 972.229924] Key type ._llcrypt registered [ 972.231566] Key type .llcrypt registered [ 972.273043] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 973.749418] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules [ 974.127556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.173285] alg: No test for adler32 (adler32-zlib) [ 975.026563] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 975.121615] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 976.703129] Key type lgssc registered [ 977.118192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 977.526105] LustreError: 89778:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 984.965567] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 988.042420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 998.699702] LustreError: 89778:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 998.707930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.879106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.289766] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1004.179560] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:38:08 (1783147088) [ 1005.709053] LustreError: 90834:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1005.714184] LustreError: 90834:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1005.717726] LustreError: 90834:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1005.838243] Lustre: server umount lustre-MDT0000 complete [ 1010.202705] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1011.192287] Key type lgssc unregistered [ 1011.312320] LNet: 92326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.315789] LNetError: 92326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.326339] LNet: Removed LNI 192.168.204.105@tcp [ 1011.641034] Key type .llcrypt unregistered [ 1011.642685] Key type ._llcrypt unregistered [ 1018.760215] Key type ._llcrypt registered [ 1018.761470] Key type .llcrypt registered [ 1018.798818] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1019.105382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.112410] alg: No test for adler32 (adler32-zlib) [ 1019.950485] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1020.029408] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1021.607114] Key type lgssc registered [ 1021.956279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.397164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.403371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.555779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.855101] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1029.642827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.855921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.149511] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1033.615316] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.521259] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.821165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.052519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.986777] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1042.374546] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1074.615840] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:39:19 (1783147159) [ 1075.205414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1075.840589] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:39:20 (1783147160) [ 1081.314446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1081.315111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.315892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.324129] Lustre: Skipped 1 previous similar message [ 1083.038123] Lustre: server umount lustre-MDT0000 complete [ 1084.278296] LustreError: 93501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147169 with bad export cookie 7942817701896007622 [ 1084.282206] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.285251] LustreError: 93501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.406543] Lustre: server umount lustre-MDT0001 complete [ 1096.063556] Lustre: server umount lustre-OST0000 complete [ 1099.781601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.929093] LustreError: 97808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.934669] LustreError: 97808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1099.980422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.266168] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1104.163679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.388300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.835879] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1108.270623] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1110.121770] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.506884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.593267] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1114.595029] LustreError: 99894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.597743] LustreError: 97798:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1114.613972] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1114.617147] LustreError: 99888:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1114.619413] LustreError: 99888:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1114.621107] LustreError: 99888:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1114.645413] Lustre: server umount lustre-OST0000 complete [ 1114.647195] LustreError: 99888:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1117.504445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1117.569619] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1117.574389] LustreError: 97797:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1117.585891] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1117.590208] LustreError: 100355:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1117.592336] LustreError: 100355:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1117.594047] LustreError: 100355:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1117.616426] Lustre: server umount lustre-OST0000 complete [ 1117.618037] LustreError: 100355:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1120.817183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.902270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1120.905144] Lustre: Found index 0 for lustre-OST0000, updating log [ 1120.908413] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1120.943208] Lustre: lustre-OST0000: new disk, initializing [ 1120.945303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1120.948527] Lustre: 100886:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1121.053204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.144191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1124.493448] Lustre: Failing over lustre-OST0000 [ 1124.552956] Lustre: server umount lustre-OST0000 complete [ 1129.951860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.955424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.960630] Lustre: Skipped 1 previous similar message [ 1129.962583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.964771] Lustre: Skipped 2 previous similar messages [ 1130.976058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.976345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.096247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.099267] Lustre: Skipped 2 previous similar messages [ 1140.191178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1140.193594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.198294] Lustre: Skipped 1 previous similar message [ 1140.253630] Lustre: server umount lustre-MDT0000 complete [ 1141.550424] LustreError: 97789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147226 with bad export cookie 7942817701896010352 [ 1141.553401] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.555098] LustreError: 97789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1141.693860] Lustre: server umount lustre-MDT0001 complete [ 1144.566400] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:40:29 (1783147229) [ 1149.432120] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1152.756836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.873353] LustreError: 103225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.896634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.141537] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1156.732832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.124291] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1158.989518] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1160.989873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1163.004050] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1165.705517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.739637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1165.772321] Lustre: lustre-OST0001: new disk, initializing [ 1165.774343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1165.777455] Lustre: 105541:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1166.120992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1166.124203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1166.160578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1167.724603] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1170.812762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1176.479776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1178.994299] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 1181.664656] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1181.667310] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.022365] Lustre: server umount lustre-MDT0001 complete [ 1184.105542] LustreError: 103221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.235637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.267905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1187.270139] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1187.283894] Lustre: 103215:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1187.572672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.575959] Lustre: Skipped 3 previous similar messages [ 1188.708850] LustreError: 107256:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1188.711390] LustreError: 107256:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1188.714774] LustreError: 103218:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1188.744597] LustreError: 107270:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.105@tcp: cfg command failed: rc = -17 [ 1188.748915] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1188.748915] [ 1188.843598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1191.885441] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:41:16 (1783147276) [ 1192.932638] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1197.296529] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1203.100411] Lustre: 109154:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1211.291517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1212.560967] Lustre: 110285:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1258.975513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1258.980368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.985606] Lustre: Skipped 2 previous similar messages [ 1258.987952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.990416] Lustre: Skipped 4 previous similar messages [ 1264.537228] Lustre: server umount lustre-MDT0000 complete [ 1264.608639] LustreError: 103963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.615883] LustreError: 103963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1265.777602] LustreError: 103207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147351 with bad export cookie 7942817701896011290 [ 1265.778958] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.782266] LustreError: 103207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1265.892605] Lustre: server umount lustre-MDT0001 complete [ 1276.932078] Lustre: server umount lustre-OST0000 complete [ 1288.410277] Lustre: server umount lustre-OST0001 complete [ 1290.185676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1293.313426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1296.126199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1299.258071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1305.761973] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1309.356047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.365528] Lustre: lustre-MDT0000: reset Object Index mappings [ 1309.514722] LustreError: 114207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.538299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.948735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1313.808092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.816672] Lustre: lustre-MDT0001: reset Object Index mappings [ 1315.504049] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1316.481395] Lustre: 115355:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1318.694841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.977466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1320.792953] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1323.552653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1324.842689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1324.843213] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1324.843437] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1325.680800] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1328.817086] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1330.034525] Lustre: 117232:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1345.319863] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:43:50 (1783147430) [ 1375.712481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.716046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.721494] Lustre: Skipped 3 previous similar messages [ 1375.723840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.726532] Lustre: Skipped 4 previous similar messages [ 1380.653843] Lustre: server umount lustre-MDT0000 complete [ 1381.347949] LustreError: 114204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.352066] LustreError: 114204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1381.886515] LustreError: 116543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147467 with bad export cookie 7942817701896058015 [ 1381.891807] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.295305] Lustre: server umount lustre-MDT0001 complete [ 1395.655351] Lustre: server umount lustre-OST0000 complete [ 1403.101377] Lustre: server umount lustre-OST0001 complete [ 1405.183483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 1408.270487] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1420.424396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.518435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1420.531393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1420.576361] Lustre: lustre-MDT0000: new disk, initializing [ 1420.604826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1421.938903] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1424.937206] Lustre: server umount lustre-MDT0000 complete [ 1428.067036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.167140] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1428.172327] Lustre: 121489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1428.179402] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1429.657917] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1431.354449] Lustre: server umount lustre-MDT0000 complete [ 1434.327453] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:45:19 (1783147519) [ 1434.950834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1435.585046] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:45:20 (1783147520) [ 1441.364753] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1444.651698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.780111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.782447] Lustre: Skipped 5 previous similar messages [ 1446.145866] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1449.028143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.069150] Lustre: 123535:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1449.093963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1449.096941] Lustre: Skipped 1 previous similar message [ 1449.134802] Lustre: lustre-MDT0001: new disk, initializing [ 1449.185070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1449.191715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1450.589487] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1452.839550] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1455.025717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.136962] Lustre: lustre-OST0000: new disk, initializing [ 1455.139447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1455.143165] Lustre: 125117:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1455.865494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1455.870300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1455.905586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1457.137983] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1461.185967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1461.252385] Lustre: 125988:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1462.981838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1462.987295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1463.015254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1463.155344] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1467.636953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1468.861078] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1586.386851] Lustre: 132360:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1586.389405] Lustre: 132360:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1586.391287] Lustre: 132360:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1586.393445] Lustre: 132360:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1586.395552] Lustre: 132360:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1586.397488] Lustre: 132360:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1667.058207] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:49:11 (1783147751) [ 1679.839841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1679.842571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.846919] Lustre: Skipped 5 previous similar messages [ 1679.848838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.850607] Lustre: Skipped 6 previous similar messages [ 1684.729877] Lustre: server umount lustre-MDT0000 complete [ 1685.950705] LustreError: 125116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147771 with bad export cookie 7942817701896111768 [ 1685.954113] LustreError: 125116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1685.955545] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.544244] LustreError: 123543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.551081] LustreError: 123543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1700.319085] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1700.436922] Lustre: server umount lustre-MDT0001 complete [ 1707.783650] Lustre: server umount lustre-OST0000 complete [ 1723.359102] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1723.421735] Lustre: server umount lustre-OST0001 complete [ 1725.546506] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 1728.851696] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1740.303539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.384770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1740.394929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1740.396557] Lustre: Skipped 1 previous similar message [ 1740.430813] Lustre: lustre-MDT0000: new disk, initializing [ 1740.432222] Lustre: Skipped 1 previous similar message [ 1740.452149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.454300] Lustre: Skipped 3 previous similar messages [ 1740.458833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1741.775351] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1745.761067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.804218] Lustre: 135373:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.910791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1747.187955] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1750.666476] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1752.402064] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1754.583036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1754.691602] Lustre: 137340:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1756.473059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1756.475476] Lustre: Skipped 1 previous similar message [ 1756.477643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1756.510276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1756.574197] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1760.113708] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1763.115505] Lustre: lustre-MDD0000: changelog on [ 1764.006846] Lustre: lustre-MDD0001: changelog on [ 2159.204450] hrtimer: interrupt took 4918337 ns [ 2188.284088] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 2205.471376] Lustre: lustre-MDD0001: changelog off [ 2213.530030] Lustre: lustre-MDD0000: changelog off [ 2237.919873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2237.932528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.932538] Lustre: Skipped 5 previous similar messages [ 2237.933108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.933112] Lustre: Skipped 10 previous similar messages [ 2238.007240] LustreError: Skipped 1 previous similar message [ 2240.330231] Lustre: server umount lustre-OST0000 complete [ 2253.811778] Lustre: server umount lustre-MDT0000 complete [ 2258.084374] LustreError: 136166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148343 with bad export cookie 7942817701896914626 [ 2258.084640] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.098972] LustreError: 136166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2258.645378] Lustre: server umount lustre-MDT0001 complete [ 2271.582588] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2275.054357] Key type lgssc unregistered [ 2275.395689] LNet: 145545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.409499] LNetError: 145545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.425321] LNet: Removed LNI 192.168.204.105@tcp [ 2276.312956] Key type .llcrypt unregistered [ 2276.315840] Key type ._llcrypt unregistered [ 2297.088272] Key type ._llcrypt registered [ 2297.095961] Key type .llcrypt registered [ 2297.211586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2312.398797] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2313.719376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.741061] alg: No test for adler32 (adler32-zlib) [ 2314.759805] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 2314.977345] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2316.655170] Key type lgssc registered [ 2317.628678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2352.652985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.671947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.957175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2354.015583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2354.081491] Lustre: lustre-MDT0000: new disk, initializing [ 2354.161364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.181625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2359.272592] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2370.635143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.818241] Lustre: 148745:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2370.874333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2370.881110] Lustre: Skipped 1 previous similar message [ 2371.003561] Lustre: lustre-MDT0001: new disk, initializing [ 2371.142887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.191940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2371.206545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2376.083377] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2385.560384] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2391.408629] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2397.943949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.247410] Lustre: lustre-OST0000: new disk, initializing [ 2398.251810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2398.263946] Lustre: 150728:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2398.359086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.113484] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2408.011688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2408.024136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2408.171877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2412.700574] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2418.818551] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_module obdecho/obdecho [ 2420.759734] LustreError: 151730:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2420.774511] LustreError: 151730:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2422.513948] LustreError: 151826:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2434.021259] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.065414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.080815] Lustre: Skipped 1 previous similar message [ 2434.104282] Lustre: Skipped 2 previous similar messages [ 2439.139222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.151992] Lustre: Skipped 2 previous similar messages [ 2439.642105] Lustre: server umount lustre-MDT0000 complete [ 2443.596447] LustreError: 148738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148528 with bad export cookie 10999953111297129849 [ 2443.602804] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2443.615157] LustreError: 148738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2444.016539] Lustre: server umount lustre-MDT0001 complete [ 2458.730607] Lustre: server umount lustre-OST0000 complete [ 2481.715614] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2494.744494] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2538.396517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.683582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2538.731168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2538.821869] Lustre: lustre-MDT0000: new disk, initializing [ 2538.889120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.934520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2543.889585] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2553.639506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.748684] Lustre: 155546:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2553.829986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2553.837320] Lustre: Skipped 1 previous similar message [ 2553.914899] Lustre: lustre-MDT0001: new disk, initializing [ 2554.048966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.090329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2554.104317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2558.080533] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2567.305346] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2573.747964] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2581.434992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.618626] Lustre: lustre-OST0000: new disk, initializing [ 2581.623327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2581.629267] Lustre: 157515:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2581.716477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.324031] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2589.322503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2589.345906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2589.548734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2599.368466] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2605.900223] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2606.750065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2614.435483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2614.750731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2620.386123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.398200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.412949] Lustre: Skipped 1 previous similar message [ 2620.430508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2623.554474] Lustre: server umount lustre-OST0000 complete [ 2630.626356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.644907] Lustre: Skipped 1 previous similar message [ 2630.654344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.668689] Lustre: Skipped 2 previous similar messages [ 2634.947993] Lustre: server umount lustre-MDT0000 complete [ 2639.783249] LustreError: 155537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148725 with bad export cookie 10999953111297130808 [ 2639.790377] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.817377] LustreError: 155537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2640.272976] Lustre: server umount lustre-MDT0001 complete [ 2644.499329] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:05:28 (1783148728) [ 2646.530252] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2648.800325] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:05:32 (1783148732) [ 2656.106188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.589461] LustreError: 159927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.615784] LustreError: 159927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2656.710089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.376150] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2661.858523] LustreError: 159927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.954191] LustreError: 159928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.772859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.764248] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2678.282351] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2682.059519] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2686.346935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.711589] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2694.905710] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2709.984706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.991297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.000220] Lustre: Skipped 1 previous similar message [ 2710.003707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.199769] Lustre: server umount lustre-MDT0000 complete [ 2716.640287] LustreError: 159927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.481438] LustreError: 159907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148803 with bad export cookie 10999953111297131690 [ 2718.484411] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.487260] LustreError: 159907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2721.130103] LustreError: 159922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.140072] LustreError: 159922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2721.149127] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 2721.152582] Lustre: Skipped 5 previous similar messages [ 2721.763742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.775398] Lustre: Skipped 2 previous similar messages [ 2724.723379] Lustre: server umount lustre-MDT0001 complete [ 2728.855791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2737.391125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.575650] LustreError: 163784:0:(llog.c:1646:llog_backup()) MGC192.168.204.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2737.581201] Lustre: 163784:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2741.484315] LustreError: 163789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.499141] LustreError: 163789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2757.994534] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.017446] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 2792.811413] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.823549] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 2798.559403] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2798.564490] Lustre: 163784:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2839.081282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.086304] Lustre: Skipped 2 previous similar messages [ 2841.336828] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2843.108551] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2845.792443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.324072] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.334014] LustreError: 163791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 2863.590445] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x98a7af1310d19de7 [ 2863.594498] Lustre: MGC192.168.204.105@tcp: Connection restored to 0@lo (at 0@lo) [ 2865.420398] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2868.315829] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.012417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2869.220922] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2871.149857] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2873.856876] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2879.458196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.458419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2879.460180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2879.460186] Lustre: Skipped 2 previous similar messages [ 2879.464496] Lustre: Skipped 1 previous similar message [ 2885.593558] Lustre: server umount lustre-MDT0000 complete [ 2886.969146] LustreError: 161766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148972 with bad export cookie 10999953111297138151 [ 2886.972292] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.972798] LustreError: 161766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2887.110150] Lustre: server umount lustre-MDT0001 complete [ 2897.753939] Lustre: server umount lustre-OST0000 complete [ 2904.889331] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2908.348136] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2921.111108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.195620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2921.206744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2921.243101] Lustre: lustre-MDT0000: new disk, initializing [ 2921.265161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.267054] Lustre: Skipped 1 previous similar message [ 2921.271782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.750616] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2927.088457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.133384] Lustre: 168861:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.251356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2928.697700] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2932.707196] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2934.687630] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2937.129978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.248869] Lustre: 170831:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2939.357535] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2942.465163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2942.502929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2943.168788] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2944.744861] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2944.822740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2946.332393] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2946.407958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2947.554678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.555439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.555940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2947.555943] Lustre: Skipped 5 previous similar messages [ 2947.558813] Lustre: Skipped 2 previous similar messages [ 2947.560991] LustreError: Skipped 1 previous similar message [ 2949.134048] Lustre: server umount lustre-OST0000 complete [ 2956.505662] Lustre: server umount lustre-MDT0000 complete [ 2957.791212] LustreError: 168854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149043 with bad export cookie 10999953111297154972 [ 2957.792135] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.795604] LustreError: 168854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2957.921494] Lustre: server umount lustre-MDT0001 complete [ 2961.059693] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:10:45 (1783149045) [ 2963.672451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.777369] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2970.013764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.308410] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2975.867352] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.706446] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.792203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.440268] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2987.075129] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3000.153140] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:11:24 (1783149084) [ 3004.895869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.898952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.903942] Lustre: Skipped 2 previous similar messages [ 3007.181905] Lustre: server umount lustre-OST0000 complete [ 3014.548284] Lustre: server umount lustre-MDT0000 complete [ 3015.757512] LustreError: 173219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149101 with bad export cookie 10999953111297155854 [ 3015.760275] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.762500] LustreError: 173219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3015.869744] Lustre: server umount lustre-MDT0001 complete [ 3019.347735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3020.340562] Key type lgssc unregistered [ 3020.472515] LNet: 176985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3020.475888] LNetError: 176985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3020.489328] LNet: Removed LNI 192.168.204.105@tcp [ 3020.811509] Key type .llcrypt unregistered [ 3020.812861] Key type ._llcrypt unregistered [ 3027.937469] Key type ._llcrypt registered [ 3027.938937] Key type .llcrypt registered [ 3027.976908] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3028.354647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3028.372928] alg: No test for adler32 (adler32-zlib) [ 3029.219815] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3029.306204] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3030.887177] Key type lgssc registered [ 3031.227194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3033.766029] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3033.771577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.247339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3054.367309] LustreError: 178204:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [ 3059.487348] LustreError: 178204:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3059.491875] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3059.494041] LustreError: MGC192.168.204.105@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3059.497370] LustreError: 178204:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3059.500174] LustreError: 178204:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 3059.502714] LustreError: 178204:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3059.519606] Lustre: server umount lustre-OST0000 complete [ 3059.521181] LustreError: 178204:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3062.100637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.221464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.349931] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3066.907731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.033458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.360087] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3070.686251] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.474836] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3074.646032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.714515] LustreError: 180514:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3074.719978] Lustre: 180514:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.105@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3074.766079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.679625] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3078.862902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3082.210217] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.211192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.215331] Lustre: Skipped 2 previous similar messages [ 3082.218381] Lustre: Skipped 1 previous similar message [ 3085.651992] Lustre: server umount lustre-MDT0000 complete [ 3086.922277] LustreError: 178684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149172 with bad export cookie 1109860500963903300 [ 3086.925711] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.927678] LustreError: 178684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3087.036026] Lustre: server umount lustre-MDT0001 complete [ 3098.519056] Lustre: server umount lustre-OST0000 complete [ 3101.451516] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3104.531261] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3116.158262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.242287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3116.252741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3116.295055] Lustre: lustre-MDT0000: new disk, initializing [ 3116.318192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.325235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3117.717887] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3121.856375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.897594] Lustre: 184036:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3121.921816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3121.923594] Lustre: Skipped 1 previous similar message [ 3121.961710] Lustre: lustre-MDT0001: new disk, initializing [ 3121.996731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.012038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3122.017174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3123.384349] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3127.144374] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3128.873979] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3131.108325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.209980] Lustre: lustre-OST0000: new disk, initializing [ 3131.211925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3131.214842] Lustre: 186000:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3131.246275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.254871] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3136.784125] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3138.040249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3138.043993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3138.071990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3138.147299] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3138.212418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3139.459655] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3139.529688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3143.136826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3143.139416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.139904] LustreError: Skipped 1 previous similar message [ 3143.140090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.140096] Lustre: Skipped 1 previous similar message [ 3143.145385] Lustre: Skipped 1 previous similar message [ 3146.189108] Lustre: server umount lustre-OST0000 complete [ 3147.743812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3147.746084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.749699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.751514] Lustre: Skipped 1 previous similar message [ 3153.376094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.379148] Lustre: Skipped 3 previous similar messages [ 3153.496421] Lustre: server umount lustre-MDT0000 complete [ 3154.735308] LustreError: 184028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149240 with bad export cookie 1109860500963903986 [ 3154.738422] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3154.738749] LustreError: 184028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3154.859478] Lustre: server umount lustre-MDT0001 complete [ 3157.571909] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:14:02 (1783149242) [ 3162.911763] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3163.856206] Key type lgssc unregistered [ 3163.967381] LNet: 188942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.969391] LNetError: 188942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.977300] LNet: Removed LNI 192.168.204.105@tcp [ 3164.259172] Key type .llcrypt unregistered [ 3164.260338] Key type ._llcrypt unregistered [ 3171.037209] Key type ._llcrypt registered [ 3171.038098] Key type .llcrypt registered [ 3171.074239] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3171.410704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.469069] alg: No test for adler32 (adler32-zlib) [ 3172.308939] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3172.388041] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3173.967154] Key type lgssc registered [ 3174.282645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3176.552580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3176.559911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.699982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.071222] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3181.977417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.108611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3183.563536] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3185.924961] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3187.722680] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3190.243944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.331988] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3190.333500] LustreError: 192002:0:(llog.c:1687:llog_backup()) MGC192.168.204.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3190.336559] Lustre: 192002:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.105@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3190.377570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.380069] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3194.875874] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3197.408207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3197.411827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.417858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.923450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.925117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.928389] Lustre: Skipped 1 previous similar message [ 3197.934515] Lustre: Skipped 1 previous similar message [ 3201.745970] Lustre: server umount lustre-MDT0000 complete [ 3202.976080] LustreError: 190119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149288 with bad export cookie 5142426535896269027 [ 3202.979550] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.980926] LustreError: 190119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.040146] LustreError: 191571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.040674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.046933] LustreError: 191571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3203.051749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3203.109657] Lustre: server umount lustre-MDT0001 complete [ 3210.512362] Lustre: server umount lustre-OST0000 complete [ 3213.638338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3216.792804] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3228.420833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.512123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3228.543137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3228.588235] Lustre: lustre-MDT0000: new disk, initializing [ 3228.613219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.622520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3229.952527] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3234.031465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.077224] Lustre: 195516:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3234.097391] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3234.098772] Lustre: Skipped 1 previous similar message [ 3234.100394] LustreError: 196300:0:(llog.c:1687:llog_backup()) MGC192.168.204.105@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3234.103154] LustreError: 196300:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3234.105053] Lustre: 196300:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3234.112963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3234.115133] Lustre: Skipped 1 previous similar message [ 3234.154269] Lustre: lustre-MDT0001: new disk, initializing [ 3234.190419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.203875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3234.207366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3235.540099] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3239.228246] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.043467] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3243.308566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.407702] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3243.409089] Lustre: Skipped 1 previous similar message [ 3243.410683] LustreError: 197461:0:(llog.c:1687:llog_backup()) MGC192.168.204.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3243.415246] LustreError: 197461:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3243.419092] Lustre: 197461:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3243.424938] Lustre: 197461:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3243.437384] Lustre: lustre-OST0000: new disk, initializing [ 3243.439760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3243.443596] Lustre: 197485:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3243.474301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.484262] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3246.582270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3246.585501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3246.617521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3249.196945] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3250.623600] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3250.696247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3252.071455] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3252.148284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3256.802302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.802834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.804400] LustreError: Skipped 1 previous similar message [ 3256.808661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.811809] Lustre: Skipped 1 previous similar message [ 3261.922224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3261.925181] Lustre: Skipped 1 previous similar message [ 3267.039130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3267.093779] Lustre: server umount lustre-OST0000 complete [ 3270.111870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3270.114041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.117420] Lustre: Skipped 1 previous similar message [ 3270.119032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.121651] Lustre: Skipped 2 previous similar messages [ 3274.589627] Lustre: server umount lustre-MDT0000 complete [ 3275.822271] LustreError: 196313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149361 with bad export cookie 5142426535896269727 [ 3275.826161] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.826330] LustreError: 196313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3275.942467] Lustre: server umount lustre-MDT0001 complete [ 3278.788053] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:16:03 (1783149363) [ 3284.296750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3288.125605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.286407] LustreError: 200616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.300864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.717554] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3292.681308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.204744] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3295.154733] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3297.355749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.368803] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3302.208748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.279649] Lustre: lustre-OST0001: new disk, initializing [ 3302.281234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3302.283574] Lustre: 202924:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3304.346387] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3309.568744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3309.571870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3309.608189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3309.730967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3317.349276] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3317.948483] Lustre: DEBUG MARKER: ADD OST3 [ 3319.537323] loop: module loaded [ 3319.556462] loop0: detected capacity change from 0 to 204800000 [ 3319.682838] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3319.686090] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3319.690098] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3321.614435] loop0: detected capacity change from 0 to 204800000 [ 3323.436533] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3323.514509] Lustre: lustre-OST0003: new disk, initializing [ 3323.516871] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3323.520227] Lustre: 204653:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3323.564250] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3323.568746] Lustre: Skipped 3 previous similar messages [ 3324.693702] Lustre: DEBUG MARKER: STOP OST3 [ 3325.138071] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3325.176976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3325.180485] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3325.192354] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3325.369353] Lustre: Failing over lustre-OST0003 [ 3325.421557] Lustre: server umount lustre-OST0003 complete [ 3325.733436] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3327.199055] LustreError: 205000:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3329.728555] Lustre: DEBUG MARKER: START OST3 again [ 3329.906911] LustreError: 203094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.532638] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3330.533479] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.538330] Lustre: Skipped 2 previous similar messages [ 3331.903239] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.985049] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3333.163991] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3333.610158] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3333.610170] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3333.808954] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3347.082914] Lustre: Failing over lustre-OST0003 [ 3347.121579] Lustre: server umount lustre-OST0003 complete [ 3347.431182] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.431861] LustreError: 202100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.437754] Lustre: Skipped 1 previous similar message [ 3347.443214] LustreError: 202100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3352.547930] LustreError: 202100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.555308] LustreError: 202100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3359.712112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3359.715034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.716571] Lustre: Skipped 2 previous similar messages [ 3362.786977] LustreError: 202099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.791649] LustreError: 202099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3365.788533] Lustre: server umount lustre-MDT0000 complete [ 3366.972498] LustreError: 200599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149452 with bad export cookie 5142426535896270595 [ 3366.978262] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.235709] Lustre: server umount lustre-MDT0001 complete [ 3374.526964] Lustre: server umount lustre-OST0000 complete [ 3375.811056] Lustre: server umount lustre-OST0001 complete [ 3377.793652] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3380.326979] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3391.716756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.797900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3391.810277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3391.843858] Lustre: lustre-MDT0000: new disk, initializing [ 3391.864175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.866574] Lustre: Skipped 1 previous similar message [ 3391.872558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3393.115424] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3397.011065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.054511] Lustre: 210065:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3397.152766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3398.409593] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3402.063404] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3403.765670] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.952361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.050780] Lustre: 212031:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3407.352754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3407.389846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3407.964896] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3411.489247] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3412.857041] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3412.923517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3414.203343] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3414.270623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3417.570422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.570737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.573112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.573119] Lustre: Skipped 8 previous similar messages [ 3417.573931] LustreError: Skipped 1 previous similar message [ 3417.586385] Lustre: Skipped 7 previous similar messages [ 3429.343080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3429.400917] Lustre: server umount lustre-OST0000 complete [ 3436.824979] Lustre: server umount lustre-MDT0000 complete [ 3438.024306] LustreError: 210055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149523 with bad export cookie 5142426535896272877 [ 3438.027651] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.028774] LustreError: 210055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3438.151575] Lustre: server umount lustre-MDT0001 complete [ 3439.275178] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:18:44 (1783149524) [ 3444.252212] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3447.421421] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3458.669812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.750433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3458.760718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3458.762380] Lustre: Skipped 3 previous similar messages [ 3458.800786] Lustre: lustre-MDT0000: new disk, initializing [ 3458.803480] Lustre: Skipped 2 previous similar messages [ 3458.828137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3458.831564] Lustre: Skipped 2 previous similar messages [ 3460.110807] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3464.163912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.207530] Lustre: 216631:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3464.312579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3465.606735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3469.205344] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3470.973884] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3473.298668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3473.403847] Lustre: 218597:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3475.167347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3475.365392] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3478.923503] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3480.263373] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3480.329069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3481.628451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3481.698615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3485.666661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.666990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3485.667628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3485.667632] Lustre: Skipped 7 previous similar messages [ 3485.671817] Lustre: Skipped 3 previous similar messages [ 3485.674270] LustreError: Skipped 1 previous similar message [ 3488.397342] Lustre: server umount lustre-OST0000 complete [ 3495.767032] Lustre: server umount lustre-MDT0000 complete [ 3495.903950] LustreError: 217432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.910700] LustreError: 217432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3496.986483] LustreError: 217426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149582 with bad export cookie 5142426535896273759 [ 3496.991254] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.118009] Lustre: server umount lustre-MDT0001 complete [ 3499.946960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.429851] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3504.129717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.560574] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3507.928865] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3509.687056] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3511.911746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.008291] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3516.354089] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3534.815802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3534.818243] LustreError: Skipped 2 previous similar messages [ 3539.219586] Lustre: server umount lustre-OST0000 complete [ 3546.520681] Lustre: server umount lustre-MDT0000 complete [ 3547.104391] LustreError: 220812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.109326] LustreError: 220812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3547.797803] LustreError: 221547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149633 with bad export cookie 5142426535896274641 [ 3547.801630] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.804097] LustreError: 221547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3547.938129] Lustre: server umount lustre-MDT0001 complete [ 3648.941709] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3649.955361] Key type lgssc unregistered [ 3650.085270] LNet: 224491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3650.087465] LNetError: 224491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3650.094285] LNet: Removed LNI 192.168.204.105@tcp [ 3650.424124] Key type .llcrypt unregistered [ 3650.425821] Key type ._llcrypt unregistered [ 3658.176422] Key type ._llcrypt registered [ 3658.177433] Key type .llcrypt registered [ 3658.214393] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3664.495156] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3664.857593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3664.902257] alg: No test for adler32 (adler32-zlib) [ 3665.753734] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3665.839427] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3667.415207] Key type lgssc registered [ 3667.767317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3679.248564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3679.259349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.347843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3680.358784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3680.401984] Lustre: lustre-MDT0000: new disk, initializing [ 3680.425618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.433194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3681.777144] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3685.840696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.881203] Lustre: 227813:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3685.905316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3685.907747] Lustre: Skipped 1 previous similar message [ 3685.948246] Lustre: lustre-MDT0001: new disk, initializing [ 3685.979608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3685.990435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3685.994093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3687.292674] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3690.869572] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3692.596247] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3694.774161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.873600] Lustre: lustre-OST0000: new disk, initializing [ 3694.875641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3694.879211] Lustre: 229782:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3694.910115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.056084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3696.060552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3696.092987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.735160] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3700.292581] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3701.652641] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3701.718314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3703.061881] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3703.129780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3706.336805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.339294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.340961] LustreError: Skipped 1 previous similar message [ 3706.341659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.344743] Lustre: Skipped 1 previous similar message [ 3711.457804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.460354] Lustre: Skipped 2 previous similar messages [ 3716.578289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.580157] Lustre: Skipped 1 previous similar message [ 3718.111112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3718.159504] Lustre: server umount lustre-OST0000 complete [ 3721.696996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.697332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.697708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3721.702191] Lustre: Skipped 1 previous similar message [ 3725.463588] Lustre: server umount lustre-MDT0000 complete [ 3726.632072] LustreError: 227808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149811 with bad export cookie 16456037749575558438 [ 3726.635597] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.635759] LustreError: 227808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3726.749114] Lustre: server umount lustre-MDT0001 complete [ 3732.430182] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3735.662056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.800844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.059619] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3739.815775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.930667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.246040] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3742.177067] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3744.306226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3746.329598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3749.033670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3749.096300] Lustre: lustre-OST0001: new disk, initializing [ 3749.097767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3749.099974] Lustre: 235016:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3749.129142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.132487] Lustre: Skipped 1 previous similar message [ 3750.941205] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3751.490185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3751.495602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3751.528561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3756.280188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3758.824274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3762.640072] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:24:07 (1783149847) [ 3765.728592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3765.732460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.736593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3765.738253] Lustre: Skipped 1 previous similar message [ 3769.501934] Lustre: server umount lustre-MDT0000 complete [ 3770.749222] LustreError: 232693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149856 with bad export cookie 16456037749575559306 [ 3770.753393] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.754425] LustreError: 232693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3770.869689] Lustre: server umount lustre-MDT0001 complete [ 3782.403219] Lustre: server umount lustre-OST0000 complete [ 3792.861631] Lustre: server umount lustre-OST0001 complete [ 3794.886357] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3798.024891] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3809.392374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.490131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.501995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3809.553502] Lustre: lustre-MDT0000: new disk, initializing [ 3809.578712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.584937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3810.987277] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3815.298765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.341747] Lustre: 238951:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3815.452884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3816.715397] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3820.318765] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3822.023614] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3824.200083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.297419] Lustre: lustre-OST0000: new disk, initializing [ 3824.298732] Lustre: Skipped 1 previous similar message [ 3824.300127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3824.301660] Lustre: Skipped 2 previous similar messages [ 3824.303351] Lustre: 240921:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3825.975229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3825.978606] Lustre: Skipped 1 previous similar message [ 3825.981371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3826.018760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3826.171599] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3829.726670] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3831.185343] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3831.258623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3832.595483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3832.662600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3836.387334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.387485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.389342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.389347] Lustre: Skipped 4 previous similar messages [ 3836.392022] Lustre: Skipped 4 previous similar messages [ 3836.404428] LustreError: Skipped 1 previous similar message [ 3839.440596] Lustre: server umount lustre-OST0000 complete [ 3840.991797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3840.993926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.806316] Lustre: server umount lustre-MDT0000 complete [ 3848.050189] LustreError: 240920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149933 with bad export cookie 16456037749575561399 [ 3848.053325] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.056806] LustreError: 240920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.180890] Lustre: server umount lustre-MDT0001 complete [ 3850.930700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.077639] LustreError: 243138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.083318] LustreError: 243138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3851.103777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.105989] Lustre: Skipped 2 previous similar messages [ 3852.348296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3855.211655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.649568] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3858.883377] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3860.624979] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3862.793644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.733704] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3867.014912] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4013.536033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.540310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.546246] Lustre: Skipped 1 previous similar message [ 4013.548899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.552105] Lustre: Skipped 6 previous similar messages [ 4017.357820] Lustre: server umount lustre-OST0000 complete [ 4024.862244] Lustre: server umount lustre-MDT0000 complete [ 4026.192208] LustreError: 243119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150111 with bad export cookie 16456037749575562281 [ 4026.195654] LustreError: 243119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4026.195680] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.306978] Lustre: server umount lustre-MDT0001 complete [ 4132.560766] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4133.622788] Key type lgssc unregistered [ 4133.755368] LNet: 246742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4133.758970] LNetError: 246742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.769309] LNet: Removed LNI 192.168.204.105@tcp [ 4134.096124] Key type .llcrypt unregistered [ 4134.097139] Key type ._llcrypt unregistered [ 4142.854301] Key type ._llcrypt registered [ 4142.855273] Key type .llcrypt registered [ 4142.898834] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4149.076915] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4149.514223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.521773] alg: No test for adler32 (adler32-zlib) [ 4150.389262] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 4150.499257] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4152.087138] Key type lgssc registered [ 4152.529893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4168.523236] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4172.240853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4172.246657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.341948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4173.361276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4173.432446] Lustre: lustre-MDT0000: new disk, initializing [ 4173.467219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.477768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4175.208272] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4179.883233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.928053] Lustre: 250782:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4179.961272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4179.963861] Lustre: Skipped 1 previous similar message [ 4180.008022] Lustre: lustre-MDT0001: new disk, initializing [ 4180.058492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.076202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4180.080680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4181.583397] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4184.040179] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4186.262500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.362796] Lustre: lustre-OST0000: new disk, initializing [ 4186.364341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4186.366651] Lustre: 252408:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4186.397539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.963295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.966643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4188.020438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4188.375368] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4192.705336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4192.766162] Lustre: lustre-OST0001: new disk, initializing [ 4192.767612] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4192.769911] Lustre: 253283:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4192.807508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4193.284672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4193.288453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4193.326050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4194.788397] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4200.253320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4205.929492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4210.656041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4210.660838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.665813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.731523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.731804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.736421] Lustre: Skipped 1 previous similar message [ 4213.738266] Lustre: Skipped 1 previous similar message [ 4214.897555] Lustre: server umount lustre-MDT0000 complete [ 4216.091212] LustreError: 254050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150301 with bad export cookie 17730066494111028965 [ 4216.095155] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.096752] LustreError: 254050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4216.219409] Lustre: server umount lustre-MDT0001 complete [ 4227.083812] Lustre: server umount lustre-OST0000 complete [ 4238.551615] Lustre: server umount lustre-OST0001 complete [ 4243.177081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4246.171370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4249.142953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4252.183535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4258.087357] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4261.458143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.467203] Lustre: lustre-MDT0000: reset Object Index mappings [ 4261.615490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.952214] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4265.688176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.694423] Lustre: lustre-MDT0001: reset Object Index mappings [ 4267.148669] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4268.079928] Lustre: 259425:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4270.246457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.371085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.373250] Lustre: Skipped 1 previous similar message [ 4271.396248] LustreError: 259778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.399684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4271.403931] LustreError: 259778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4272.246830] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4275.023291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4276.140065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4277.013152] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4280.086167] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4281.243508] Lustre: 261290:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4286.433458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.433740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.438283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.442520] Lustre: Skipped 2 previous similar messages [ 4286.445791] Lustre: Skipped 2 previous similar messages [ 4288.477135] Lustre: server umount lustre-MDT0000 complete [ 4289.642313] LustreError: 258312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150374 with bad export cookie 17730066494111035685 [ 4289.645450] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.645567] LustreError: 258312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4289.757721] Lustre: server umount lustre-MDT0001 complete [ 4300.739501] Lustre: server umount lustre-OST0000 complete [ 4312.217504] Lustre: server umount lustre-OST0001 complete [ 4318.147209] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:33:22 (1783150402) [ 4323.305371] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4326.409718] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4337.701044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.779814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.790812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4337.826641] Lustre: lustre-MDT0000: new disk, initializing [ 4337.846565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.848380] Lustre: Skipped 1 previous similar message [ 4337.853312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4339.175859] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4343.226763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.273513] Lustre: 265535:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4343.381943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4344.690769] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4348.295906] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4350.073704] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4352.286797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4352.386649] Lustre: lustre-OST0000: new disk, initializing [ 4352.388180] Lustre: Skipped 1 previous similar message [ 4352.389704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4352.391867] Lustre: Skipped 2 previous similar messages [ 4352.393970] Lustre: 267500:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4353.659288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4353.661711] Lustre: Skipped 1 previous similar message [ 4353.663766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4353.697502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4354.296932] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4357.903939] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4359.279319] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4359.351308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4360.685100] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4360.755127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4364.257859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4364.258072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.260805] LustreError: Skipped 1 previous similar message [ 4364.264323] Lustre: Skipped 2 previous similar messages [ 4364.265060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4364.270217] Lustre: Skipped 4 previous similar messages [ 4367.568852] Lustre: server umount lustre-OST0000 complete [ 4368.863768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.865883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.868972] Lustre: Skipped 1 previous similar message [ 4368.870467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.617239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.621450] Lustre: Skipped 5 previous similar messages [ 4383.199152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4383.253677] Lustre: server umount lustre-MDT0000 complete [ 4383.712446] LustreError: 265544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.716778] LustreError: 265544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4384.517655] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150469 with bad export cookie 17730066494111037764 [ 4384.520413] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4384.522196] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4384.640701] Lustre: server umount lustre-MDT0001 complete [ 4390.463078] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4393.815628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.941864] LustreError: 270429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.954916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.956698] Lustre: Skipped 2 previous similar messages [ 4395.307129] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4398.160774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.654059] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4400.592480] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4402.753364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4404.754705] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4407.573053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.649351] Lustre: lustre-OST0001: new disk, initializing [ 4407.651421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4407.654302] Lustre: 272737:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4409.473904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4409.477277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4409.516307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4409.671317] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4414.146333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4415.354318] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4417.013936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4417.017623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4419.040095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4419.042463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.045819] Lustre: Skipped 1 previous similar message [ 4419.047422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.792188] Lustre: server umount lustre-MDT0000 complete [ 4425.184589] LustreError: 270448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.188749] LustreError: 270448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4425.980171] LustreError: 270411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150511 with bad export cookie 17730066494111038646 [ 4425.984646] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.985823] LustreError: 270411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.096337] Lustre: server umount lustre-MDT0001 complete [ 4437.514917] Lustre: server umount lustre-OST0000 complete [ 4449.122259] Lustre: server umount lustre-OST0001 complete [ 4454.380828] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4458.408863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.538557] LustreError: 275784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.557238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.559221] Lustre: Skipped 3 previous similar messages [ 4459.976257] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4462.947364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.543411] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4465.515325] Lustre: 276929:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4467.658832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4468.835914] LustreError: 277282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.842697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4469.653225] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4472.412228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.525189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4474.497503] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4477.872831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4479.242445] Lustre: 278804:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4481.270521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4481.278398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4484.384740] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:36:09 (1783150569) [ 4498.911756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4498.913922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.916945] Lustre: Skipped 3 previous similar messages [ 4498.918605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.920303] Lustre: Skipped 4 previous similar messages [ 4502.049467] Lustre: server umount lustre-MDT0000 complete [ 4503.397267] LustreError: 275766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150588 with bad export cookie 17730066494111044953 [ 4503.399945] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.402793] LustreError: 275766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4504.545024] LustreError: 276519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.551759] LustreError: 276519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4509.564073] Lustre: server umount lustre-MDT0001 complete [ 4525.023099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4525.056867] Lustre: server umount lustre-OST0000 complete [ 4532.500947] Lustre: server umount lustre-OST0001 complete [ 4534.711045] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4538.611956] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4555.258128] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4558.845173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.929874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4558.940180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4558.975655] Lustre: lustre-MDT0000: new disk, initializing [ 4559.004266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4560.405162] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4564.851920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.895366] Lustre: 283030:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4565.001809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4566.387394] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4571.196546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4571.306100] Lustre: 284667:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.804445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4573.256806] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4577.449059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4578.860246] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4579.529271] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4584.066473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4589.733673] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:37:54 (1783150674) [ 4590.560327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.567589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.578899] Lustre: Skipped 5 previous similar messages [ 4590.583783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.587902] Lustre: Skipped 8 previous similar messages [ 4596.675610] Lustre: server umount lustre-MDT0000 complete [ 4598.339516] LustreError: 283827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150683 with bad export cookie 17730066494111052835 [ 4598.347221] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.347958] LustreError: 283827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.476763] Lustre: server umount lustre-MDT0001 complete [ 4610.316552] Lustre: server umount lustre-OST0000 complete [ 4622.113165] Lustre: server umount lustre-OST0001 complete [ 4625.202515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.361719] LustreError: 287778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.366920] LustreError: 287778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4625.390861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.394102] Lustre: Skipped 7 previous similar messages [ 4626.764419] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4630.179994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.874870] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4634.395800] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.337771] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4639.673349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.453883] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4645.494908] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4658.209165] loop1: detected capacity change from 0 to 204800000 [ 4658.284718] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4658.289591] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4658.293620] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4660.577540] loop1: detected capacity change from 0 to 204800000 [ 4663.222127] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.277535] LustreError: 287768:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs28181: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4663.285681] LustreError: 287768:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs28181 [ 4663.289406] LustreError: 287768:0:(mgs_handler.c:517:mgs_target_reg()) fs28181-MDT0001: Error checking target index: rc = -13 [ 4663.295556] LustreError: fs28181-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4663.301152] LustreError: 291492:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4663.306284] LustreError: 291492:0:(tgt_mount.c:2129:server_put_super()) no obd fs28181-MDT0001 [ 4663.310843] LustreError: 291492:0:(tgt_mount.c:116:server_deregister_mount()) fs28181-MDT0001 not registered [ 4663.329298] Lustre: server umount fs28181-MDT0001 complete [ 4663.331285] LustreError: 291492:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4666.863281] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.905276] Lustre: Setting parameter fs28181-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs28181-MDT0001 [ 4666.910470] Lustre: Skipped 2 previous similar messages [ 4666.929461] Lustre: srv-fs28181-MDT0001: No data found on store. Initialize space. [ 4666.931888] Lustre: Skipped 4 previous similar messages [ 4666.974266] Lustre: fs28181-MDT0001: new disk, initializing [ 4666.976050] Lustre: Skipped 3 previous similar messages [ 4667.020589] Lustre: srv-fs28181-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4667.022974] Lustre: 291959:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28181-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4667.025820] LustreError: 291959:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28181-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4668.560204] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4671.746331] Lustre: server umount fs28181-MDT0001 complete [ 4677.224913] loop1: detected capacity change from 0 to 204800000 [ 4679.658632] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.850057] Lustre: 293526:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28181-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4679.855456] LustreError: 293526:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28181-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4681.526852] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4683.425854] Lustre: server umount fs28181-MDT0001 complete [ 4691.426262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.426303] LustreError: 289604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.432842] Lustre: Skipped 4 previous similar messages [ 4691.438180] LustreError: 289604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 4693.880203] Lustre: server umount lustre-MDT0000 complete [ 4697.030728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.083403] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4698.690034] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4701.866219] loop1: detected capacity change from 0 to 204800000 [ 4702.698799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4704.257785] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.372993] Lustre: 295968:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28181-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4704.376450] LustreError: 295968:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28181-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4706.098323] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4708.134077] Lustre: server umount fs28181-MDT0001 complete [ 4717.537205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4723.169043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.171912] Lustre: Skipped 12 previous similar messages [ 4729.823094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4729.872883] Lustre: server umount lustre-OST0000 complete [ 4737.598107] Lustre: server umount lustre-MDT0000 complete [ 4738.970573] LustreError: 287759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150824 with bad export cookie 17730066494111055509 [ 4738.974618] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4738.976877] LustreError: 287759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4739.110396] Lustre: server umount lustre-MDT0001 complete [ 4742.913979] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4744.098672] Key type lgssc unregistered [ 4744.248725] LNet: 298307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4744.253929] LNetError: 298307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4744.268467] LNet: Removed LNI 192.168.204.105@tcp [ 4744.664144] Key type .llcrypt unregistered [ 4744.666955] Key type ._llcrypt unregistered [ 4753.423436] Key type ._llcrypt registered [ 4753.425034] Key type .llcrypt registered [ 4753.469129] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4753.787098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4753.814077] alg: No test for adler32 (adler32-zlib) [ 4754.674391] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 4754.771729] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4756.359158] Key type lgssc registered [ 4756.774476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4760.033516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4760.046554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.211278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.730686] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4766.357185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.498114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4767.857099] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4770.880099] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4773.017257] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4775.882514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.046762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.358419] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4781.163154] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4786.280626] LustreError: 301385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.291869] LustreError: 301385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4787.545728] LustreError: 301385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.401622] LustreError: 301384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.412394] LustreError: 301384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4796.907865] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.917423] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4797.900376] loop1: detected capacity change from 0 to 204800000 [ 4797.964834] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4797.968920] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4797.973620] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4799.870734] loop1: detected capacity change from 0 to 204800000 [ 4801.794264] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.846315] Lustre: fs17891-OST0000: new disk, initializing [ 4801.848760] Lustre: srv-fs17891-OST0000: No data found on store. Initialize space. [ 4801.851308] Lustre: 303310:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.874747] Lustre: fs17891-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.017612] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.025301] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4803.783579] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4807.247797] Lustre: server umount fs17891-OST0000 complete [ 4811.255265] loop1: detected capacity change from 0 to 204800000 [ 4812.260615] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.265367] LustreError: 301386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4813.231935] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.299546] Lustre: fs17891-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.292485] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4818.272746] loop2: detected capacity change from 0 to 204800000 [ 4818.374177] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4818.379561] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4818.384140] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4820.098770] loop2: detected capacity change from 0 to 204800000 [ 4821.968904] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4822.003908] LustreError: fs17891-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4822.007572] LustreError: fs17891-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4822.010941] LustreError: 306346:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4822.013257] LustreError: 306346:0:(tgt_mount.c:2129:server_put_super()) no obd fs17891-OST0001 [ 4822.015251] LustreError: 306346:0:(tgt_mount.c:116:server_deregister_mount()) fs17891-OST0001 not registered [ 4822.026295] Lustre: server umount fs17891-OST0001 complete [ 4822.027563] LustreError: 306346:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4825.338414] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.380789] Lustre: fs17891-OST0001: new disk, initializing [ 4825.382961] Lustre: srv-fs17891-OST0001: No data found on store. Initialize space. [ 4825.386232] Lustre: 306896:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4825.410455] Lustre: fs17891-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4827.307872] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4828.521254] LustreError: 301384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.527431] LustreError: 301384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4830.670421] Lustre: server umount fs17891-OST0000 complete [ 4833.676721] Lustre: server umount fs17891-OST0001 complete [ 4837.753748] loop1: detected capacity change from 0 to 204800000 [ 4839.600066] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4841.591984] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4843.782212] loop2: detected capacity change from 0 to 204800000 [ 4845.750889] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.816391] Lustre: fs17891-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4845.818975] Lustre: Skipped 1 previous similar message [ 4847.777928] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4850.715131] Lustre: server umount fs17891-OST0000 complete [ 4853.734717] Lustre: server umount fs17891-OST0001 complete [ 4858.340244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4858.342182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.353287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4858.356671] Lustre: Skipped 1 previous similar message [ 4863.458042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4863.458039] LustreError: 302055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.458058] LustreError: 302055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4863.460532] Lustre: Skipped 1 previous similar message [ 4863.637396] Lustre: server umount lustre-OST0000 complete [ 4868.577366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.580651] Lustre: Skipped 1 previous similar message [ 4868.582817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4868.584502] Lustre: Skipped 1 previous similar message [ 4873.696469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4873.698441] Lustre: Skipped 1 previous similar message [ 4878.816317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4878.818481] Lustre: Skipped 1 previous similar message [ 4879.327100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.394180] Lustre: server umount lustre-MDT0000 complete [ 4880.655424] LustreError: 301408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150965 with bad export cookie 2779175498308128689 [ 4880.659239] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.659993] LustreError: 301408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4880.780631] Lustre: server umount lustre-MDT0001 complete [ 4884.559932] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4885.644382] Key type lgssc unregistered [ 4885.762338] LNet: 312809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4885.765844] LNetError: 312809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4885.778365] LNet: Removed LNI 192.168.204.105@tcp [ 4886.091101] Key type .llcrypt unregistered [ 4886.092264] Key type ._llcrypt unregistered [ 4896.495645] Key type ._llcrypt registered [ 4896.496704] Key type .llcrypt registered [ 4896.532338] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4902.674932] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4903.020148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4903.046173] alg: No test for adler32 (adler32-zlib) [ 4903.893761] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 4903.979363] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4905.559106] Key type lgssc registered [ 4905.916979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4917.582442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4917.587629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.670962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4918.680241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4918.717043] Lustre: lustre-MDT0000: new disk, initializing [ 4918.737135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.742421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4920.107520] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4924.254867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.307066] Lustre: 316638:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4924.337758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4924.341380] Lustre: Skipped 1 previous similar message [ 4924.381259] Lustre: lustre-MDT0001: new disk, initializing [ 4924.415504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.429630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4924.432981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4925.791141] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4929.486532] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4931.294531] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4933.591512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.698089] Lustre: lustre-OST0000: new disk, initializing [ 4933.699945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4933.702545] Lustre: 318607:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4933.732766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.610156] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4939.256611] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4943.288643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4943.291711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4943.321312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4948.466597] Lustre: Mounted lustre-client [ 4948.794200] Lustre: Unmounted lustre-client [ 4953.569773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.572307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.573120] LustreError: Skipped 1 previous similar message [ 4953.577901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.580718] Lustre: Skipped 1 previous similar message [ 4957.774836] Lustre: server umount lustre-OST0000 complete [ 4960.223896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4960.226314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.229585] Lustre: Skipped 1 previous similar message [ 4960.231269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.808889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.809728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.082543] Lustre: server umount lustre-MDT0000 complete [ 4966.283394] LustreError: 318606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151051 with bad export cookie 4417276065532230151 [ 4966.287849] LustreError: 318606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4966.288713] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.408218] Lustre: server umount lustre-MDT0001 complete [ 4970.152743] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4971.222620] Key type lgssc unregistered [ 4995.103253] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [ 4995.935435] LNet: 320766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4995.938379] LNetError: 320766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5072.741317] LNet: Removed LNI 192.168.204.105@tcp [ 5073.009237] Key type .llcrypt unregistered [ 5073.010473] Key type ._llcrypt unregistered [ 5079.619689] Key type ._llcrypt registered [ 5079.621499] Key type .llcrypt registered [ 5079.657712] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5086.444792] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5086.835255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5086.852915] alg: No test for adler32 (adler32-zlib) [ 5087.712646] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5087.805083] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5089.383194] Key type lgssc registered [ 5089.963702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5102.708238] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:46:27 (1783151187) [ 5105.445585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5105.459664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.563539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5106.572763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5106.615322] Lustre: lustre-MDT0000: new disk, initializing [ 5106.635867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.644125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5108.307465] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5113.384343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.463880] Lustre: 323966:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5113.517412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5113.522424] Lustre: Skipped 1 previous similar message [ 5113.564850] Lustre: lustre-MDT0001: new disk, initializing [ 5113.604923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5113.623192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5113.631977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5115.170750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5119.967902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5122.652361] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5126.037359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.175878] Lustre: lustre-OST0000: new disk, initializing [ 5126.179497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5126.184325] Lustre: 325934:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5126.227597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.300597] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5132.683453] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.838189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5134.841622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5134.871169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5145.061072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5145.062063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.064403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.066400] LustreError: Skipped 1 previous similar message [ 5145.075196] Lustre: Skipped 1 previous similar message [ 5146.347497] Lustre: server umount lustre-OST0000 complete [ 5149.664361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5149.669672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.678320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5149.682113] Lustre: Skipped 1 previous similar message [ 5149.876658] Lustre: server umount lustre-MDT0000 complete [ 5151.621769] LustreError: 323961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151236 with bad export cookie 2549284651365619974 [ 5151.625785] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5151.628526] LustreError: 323961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5151.769169] Lustre: server umount lustre-MDT0001 complete [ 5156.701367] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5157.981862] Key type lgssc unregistered [ 5158.116366] LNet: 327844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5158.118668] LNetError: 327844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5158.130288] LNet: Removed LNI 192.168.204.105@tcp [ 5158.451585] Key type .llcrypt unregistered [ 5158.453227] Key type ._llcrypt unregistered [ 5166.203322] Key type ._llcrypt registered [ 5166.204326] Key type .llcrypt registered [ 5166.235329] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5166.621600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5166.638123] alg: No test for adler32 (adler32-zlib) [ 5167.485690] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5167.571940] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5169.151123] Key type lgssc registered [ 5169.556049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5172.556276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5172.564538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5173.744581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.572970] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5179.781108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.998744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.796064] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5185.345668] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5187.866954] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5190.373152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5190.525834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.168903] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5195.736763] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5209.456774] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.032572] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5212.530255] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 5215.204219] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.206139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5215.210695] Lustre: Skipped 2 previous similar messages [ 5215.216651] Lustre: Skipped 2 previous similar messages [ 5217.641360] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 5217.714394] Lustre: server umount lustre-MDT0000 complete [ 5219.331193] LustreError: 329076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151304 with bad export cookie 7812772092395169987 [ 5219.333427] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.335286] LustreError: 329076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5220.319857] LustreError: 331569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.320078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.327037] LustreError: 331569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5220.331578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5222.761846] LustreError: 329088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.352255] LustreError: 331584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.352696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5224.366718] Lustre: Skipped 1 previous similar message [ 5225.532031] Lustre: server umount lustre-MDT0001 complete [ 5241.311171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5241.392172] Lustre: server umount lustre-OST0000 complete [ 5244.786195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.563773] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5247.742941] Lustre: 333292:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.105@tcp [ 5248.609143] Lustre: server umount MGS complete [ 5252.532511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.675274] LustreError: 333863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.695682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.463786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5257.696093] LustreError: 333864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.011936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.146270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.880228] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5263.748186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5266.403577] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5269.093860] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.801636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5272.225126] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5280.090398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5282.273877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.276581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.280169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.292435] Lustre: Skipped 1 previous similar message [ 5283.501323] Lustre: server umount lustre-OST0000 complete [ 5287.070587] Lustre: server umount lustre-MDT0000 complete [ 5287.393900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.395142] LustreError: 333859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.401900] Lustre: Skipped 1 previous similar message [ 5287.414783] LustreError: 333859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5288.740437] LustreError: 333843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151374 with bad export cookie 7812772092395170813 [ 5288.744526] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.746276] LustreError: 333843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5288.866776] Lustre: server umount lustre-MDT0001 complete [ 5293.273105] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5294.830500] Key type lgssc unregistered [ 5295.003694] LNet: 337299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5295.009371] LNetError: 337299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5295.020531] LNet: Removed LNI 192.168.204.105@tcp [ 5295.487636] Key type .llcrypt unregistered [ 5295.490230] Key type ._llcrypt unregistered [ 5303.889539] Key type ._llcrypt registered [ 5303.890641] Key type .llcrypt registered [ 5303.936519] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5310.987457] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5311.349459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5311.394790] alg: No test for adler32 (adler32-zlib) [ 5312.264445] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5312.364605] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5313.951195] Key type lgssc registered [ 5314.514924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.046068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.058423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.157816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5332.170922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5332.211950] Lustre: lustre-MDT0000: new disk, initializing [ 5332.235543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.242388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5333.904073] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5339.620636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.699267] Lustre: 340432:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5339.743459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5339.747439] Lustre: Skipped 1 previous similar message [ 5339.800062] Lustre: lustre-MDT0001: new disk, initializing [ 5339.843767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.869170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5339.882221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5341.613648] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5346.497660] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5348.269166] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5350.793851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.950643] Lustre: lustre-OST0000: new disk, initializing [ 5350.952750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5350.955848] Lustre: 342401:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5350.993233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.303915] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5355.020563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5355.027076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5355.083115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5357.811930] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5359.404122] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5359.514538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5361.294481] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5361.397891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5365.219513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5365.222333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.225020] LustreError: Skipped 1 previous similar message [ 5365.235913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5365.242328] Lustre: Skipped 1 previous similar message [ 5368.411265] Lustre: server umount lustre-OST0000 complete [ 5370.338140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.339928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.346202] Lustre: Skipped 2 previous similar messages [ 5370.353832] Lustre: Skipped 1 previous similar message [ 5375.457241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5375.462587] Lustre: Skipped 1 previous similar message [ 5376.122450] Lustre: server umount lustre-MDT0000 complete [ 5377.766534] LustreError: 340424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151463 with bad export cookie 5156248912820015463 [ 5377.772825] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5377.775360] LustreError: 340424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5377.913412] Lustre: server umount lustre-MDT0001 complete [ 5380.961610] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:51:05 (1783151465) [ 5388.150428] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5391.824799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5406.690134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.793285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5406.804553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5406.841712] Lustre: lustre-MDT0000: new disk, initializing [ 5406.863355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.869648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5408.462358] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5412.596703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.636447] Lustre: 347153:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.658665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5412.660724] Lustre: Skipped 1 previous similar message [ 5412.698519] Lustre: lustre-MDT0001: new disk, initializing [ 5412.727146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.737557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5412.740380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5413.980368] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5417.454717] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5419.122916] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5421.248874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.348846] Lustre: lustre-OST0000: new disk, initializing [ 5421.351306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5421.354535] Lustre: 349119:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5421.382535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.147159] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5426.635955] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5431.798141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5431.802828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5431.829698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.019155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5442.021996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.026503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5442.033432] Lustre: Skipped 1 previous similar message [ 5442.039881] Lustre: Skipped 1 previous similar message [ 5447.141558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5447.145560] Lustre: Skipped 1 previous similar message [ 5456.351109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5456.411220] Lustre: server umount lustre-OST0000 complete [ 5458.912314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5458.916346] LustreError: Skipped 1 previous similar message [ 5458.918594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.925393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5458.928432] Lustre: Skipped 2 previous similar messages [ 5472.223185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5472.322215] Lustre: server umount lustre-MDT0000 complete [ 5473.610120] LustreError: 347948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151558 with bad export cookie 5156248912820016345 [ 5473.613223] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5473.613445] LustreError: 347948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5473.732864] Lustre: server umount lustre-MDT0001 complete [ 5477.836655] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5479.132959] Key type lgssc unregistered [ 5479.293541] LNet: 351272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5479.297715] LNetError: 351272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5479.307587] LNet: Removed LNI 192.168.204.105@tcp [ 5479.771682] Key type .llcrypt unregistered [ 5479.773981] Key type ._llcrypt unregistered [ 5519.506578] Key type ._llcrypt registered [ 5519.507708] Key type .llcrypt registered [ 5519.545358] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5519.659234] Key type .llcrypt unregistered [ 5519.660340] Key type ._llcrypt unregistered [ 5551.067972] Key type ._llcrypt registered [ 5551.069189] Key type .llcrypt registered [ 5551.106179] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5551.200311] Key type .llcrypt unregistered [ 5551.201734] Key type ._llcrypt unregistered [ 5562.778889] Key type ._llcrypt registered [ 5562.780121] Key type .llcrypt registered [ 5562.816630] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5562.930205] Key type .llcrypt unregistered [ 5562.932150] Key type ._llcrypt unregistered [ 5575.164229] Key type ._llcrypt registered [ 5575.165375] Key type .llcrypt registered [ 5575.204734] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5575.315182] Key type .llcrypt unregistered [ 5575.316210] Key type ._llcrypt unregistered [ 5607.839497] Key type ._llcrypt registered [ 5607.840930] Key type .llcrypt registered [ 5607.876894] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5607.974186] Key type .llcrypt unregistered [ 5607.975372] Key type ._llcrypt unregistered [ 5622.160440] Key type ._llcrypt registered [ 5622.161667] Key type .llcrypt registered [ 5622.198848] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5622.295447] Key type .llcrypt unregistered [ 5622.297063] Key type ._llcrypt unregistered [ 5634.425804] Key type ._llcrypt registered [ 5634.427128] Key type .llcrypt registered [ 5634.462089] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5634.555476] Key type .llcrypt unregistered [ 5634.556625] Key type ._llcrypt unregistered [ 5646.784410] Key type ._llcrypt registered [ 5646.785397] Key type .llcrypt registered [ 5646.821279] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5646.912395] Key type .llcrypt unregistered [ 5646.913983] Key type ._llcrypt unregistered [ 5660.812541] Key type ._llcrypt registered [ 5660.813511] Key type .llcrypt registered [ 5660.848283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5660.939276] Key type .llcrypt unregistered [ 5660.940271] Key type ._llcrypt unregistered [ 5672.938076] Key type ._llcrypt registered [ 5672.939323] Key type .llcrypt registered [ 5672.974579] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5673.066316] Key type .llcrypt unregistered [ 5673.067325] Key type ._llcrypt unregistered [ 5693.943214] Key type ._llcrypt registered [ 5693.944203] Key type .llcrypt registered [ 5693.976206] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5694.059214] Key type .llcrypt unregistered [ 5694.060210] Key type ._llcrypt unregistered [ 5708.165231] Key type ._llcrypt registered [ 5708.166563] Key type .llcrypt registered [ 5708.204370] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5708.305726] Key type .llcrypt unregistered [ 5708.307521] Key type ._llcrypt unregistered [ 5721.338351] Key type ._llcrypt registered [ 5721.339396] Key type .llcrypt registered [ 5721.386117] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5721.516432] Key type .llcrypt unregistered [ 5721.518024] Key type ._llcrypt unregistered [ 5742.384429] Key type ._llcrypt registered [ 5742.386161] Key type .llcrypt registered [ 5742.433182] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5749.868146] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5750.196367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5750.222477] alg: No test for adler32 (adler32-zlib) [ 5751.074549] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5751.158317] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5752.735183] Key type lgssc registered [ 5753.229762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5764.048957] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5764.055180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.134697] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5765.144731] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5765.181809] Lustre: tst250fs-MDT0000: new disk, initializing [ 5765.203410] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.208834] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5766.498376] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5770.522044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.561770] Lustre: 371181:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5770.584634] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5770.586511] Lustre: Skipped 1 previous similar message [ 5770.622914] Lustre: tst250fs-MDT0001: new disk, initializing [ 5770.654554] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5770.665760] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5770.670218] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5771.933643] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5774.114796] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5776.166102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.266831] Lustre: tst250fs-OST0000: new disk, initializing [ 5776.269090] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5776.272339] Lustre: 372816:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5776.302429] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.130897] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5782.004480] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5782.007706] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5782.050718] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5782.077392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.138505] Lustre: tst250fs-OST0001: new disk, initializing [ 5782.140275] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5782.143214] Lustre: 373688:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5782.177145] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.174924] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5786.898609] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5788.530050] Lustre: 371186:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.105@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 5797.857446] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.859051] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.863380] Lustre: Skipped 3 previous similar messages [ 5797.865071] Lustre: Skipped 2 previous similar messages [ 5802.774961] Lustre: server umount tst250fs-MDT0000 complete [ 5803.920020] LustreError: 371175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151889 with bad export cookie 15568631513403914906 [ 5803.923872] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.923942] LustreError: 371175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5804.036765] Lustre: server umount tst250fs-MDT0001 complete [ 5815.298320] Lustre: server umount tst250fs-OST0000 complete [ 5826.717818] Lustre: server umount tst250fs-OST0001 complete [ 5832.564946] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5835.245199] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5845.229243] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:58:49 (1783151929) [ 5845.781542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5851.736831] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5855.526475] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5866.871238] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5754 sec ========== 03:59:11 (1783151951) [ 5867.444134] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:59:12 (1783151952) === [ 5868.585811] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:59:13 (1783151953) === [ 5883.269470] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5884.348213] Key type lgssc unregistered [ 5884.481299] LNet: 382786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5884.484333] LNetError: 382786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5884.492369] LNet: Removed LNI 192.168.204.105@tcp [ 5884.893118] Key type .llcrypt unregistered [ 5884.894515] Key type ._llcrypt unregistered