[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481931315 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007451] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021808] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026811] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030100] *** VALIDATE tmpfs *** [ 0.032168] *** VALIDATE proc *** [ 0.033136] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042195] debug: unmapping init [mem 0xffffffffa5659000-0xffffffffa5660fff] [ 0.044931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045652] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.054209] rcu: Hierarchical SRCU implementation. [ 0.056428] smp: Bringing up secondary CPUs ... [ 0.057539] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062019] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240025] node 0 deferred pages initialised in 172ms [ 0.242564] devtmpfs: initialized [ 0.243209] x86/mm: Memory block size: 128MB [ 0.245691] gcov: version magic: 0x41383552 [ 0.247107] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.248126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249260] pinctrl core: initialized pinctrl subsystem [ 0.250342] [ 0.251000] ************************************************************* [ 0.254022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256015] ** ** [ 0.259013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.261012] ** ** [ 0.263013] ** This means that this kernel is built to expose internal ** [ 0.266015] ** IOMMU data structures, which may compromise security on ** [ 0.268014] ** your system. ** [ 0.271014] ** ** [ 0.273013] ** If you see this message and you are not debugging the ** [ 0.275014] ** kernel, report this immediately to your vendor! ** [ 0.278015] ** ** [ 0.280020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.282014] ************************************************************* [ 0.286942] NET: Registered protocol family 16 [ 0.289617] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.300167] cpuidle: using governor menu [ 0.302932] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.305757] PCI: Using configuration type 1 for base access [ 0.308176] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317134] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322140] cryptd: max_cpu_qlen set to 1000 [ 0.326264] ACPI: Added _OSI(Module Device) [ 0.327022] ACPI: Added _OSI(Processor Device) [ 0.329020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334212] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343631] ACPI: Interpreter enabled [ 0.344058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.346013] ACPI: Using IOAPIC for interrupt routing [ 0.347121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.362764] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.365039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.367016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377430] acpiphp: Slot [2] registered [ 0.379161] acpiphp: Slot [5] registered [ 0.380277] acpiphp: Slot [6] registered [ 0.382141] acpiphp: Slot [7] registered [ 0.383137] acpiphp: Slot [8] registered [ 0.385157] acpiphp: Slot [9] registered [ 0.386122] acpiphp: Slot [10] registered [ 0.388143] acpiphp: Slot [3] registered [ 0.389082] acpiphp: Slot [4] registered [ 0.391093] acpiphp: Slot [11] registered [ 0.392109] acpiphp: Slot [12] registered [ 0.393103] acpiphp: Slot [13] registered [ 0.395094] acpiphp: Slot [14] registered [ 0.396106] acpiphp: Slot [15] registered [ 0.397085] acpiphp: Slot [16] registered [ 0.399095] acpiphp: Slot [17] registered [ 0.400101] acpiphp: Slot [18] registered [ 0.402120] acpiphp: Slot [19] registered [ 0.404104] acpiphp: Slot [20] registered [ 0.405134] acpiphp: Slot [21] registered [ 0.407165] acpiphp: Slot [22] registered [ 0.409100] acpiphp: Slot [23] registered [ 0.410117] acpiphp: Slot [24] registered [ 0.412152] acpiphp: Slot [25] registered [ 0.414115] acpiphp: Slot [26] registered [ 0.416106] acpiphp: Slot [27] registered [ 0.417161] acpiphp: Slot [28] registered [ 0.419162] acpiphp: Slot [29] registered [ 0.421124] acpiphp: Slot [30] registered [ 0.422096] acpiphp: Slot [31] registered [ 0.424059] PCI host bridge to bus 0000:00 [ 0.425015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.427020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.429019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.432022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.435023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.437021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.443045] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445861] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.455795] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.462034] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.468019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473527] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481682] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506941] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.512197] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.526016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.545015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555878] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.570025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.585026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596106] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.633038] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.642000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.649016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.655023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677060] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.688023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.693022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.731877] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.745015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.753024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.785018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.800999] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.802000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.803391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.806397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.808231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.813107] iommu: Default domain type: Passthrough [ 0.816376] SCSI subsystem initialized [ 0.817125] ACPI: bus type USB registered [ 0.819117] usbcore: registered new interface driver usbfs [ 0.821097] usbcore: registered new interface driver hub [ 0.822059] usbcore: registered new device driver usb [ 0.824184] pps_core: LinuxPPS API ver. 1 registered [ 0.826011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.830055] PTP clock support registered [ 0.832209] EDAC MC: Ver: 3.0.0 [ 0.834158] PCI: Using ACPI for IRQ routing [ 0.836887] NetLabel: Initializing [ 0.838013] NetLabel: domain hash size = 128 [ 0.840015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.842061] NetLabel: unlabeled traffic allowed by default [ 0.843117] vgaarb: loaded [ 0.845386] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.847018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.853037] clocksource: Switched to clocksource kvm-clock [ 0.962662] VFS: Disk quotas dquot_6.6.0 [ 0.964197] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.966685] *** VALIDATE ramfs *** [ 0.968630] *** VALIDATE hugetlbfs *** [ 0.970081] pnp: PnP ACPI init [ 0.972422] pnp: PnP ACPI: found 6 devices [ 0.988665] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.992402] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994921] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.997513] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.000524] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.003339] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.006625] NET: Registered protocol family 2 [ 1.009543] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.015103] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.019255] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.025614] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029335] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032882] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035907] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038764] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042278] NET: Registered protocol family 1 [ 1.044832] RPC: Registered named UNIX socket transport module. [ 1.047495] RPC: Registered udp transport module. [ 1.049137] RPC: Registered tcp transport module. [ 1.050633] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.053127] NET: Registered protocol family 44 [ 1.054957] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.057320] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.059956] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.063038] PCI: CLS 0 bytes, default 64 [ 1.065706] Unpacking initramfs... [ 2.523194] debug: unmapping init [mem 0xffff9471fcc54000-0xffff9471fffbffff] [ 2.527097] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.529333] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.532285] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.032571] Initialise system trusted keyrings [ 3.035668] Key type blacklist registered [ 3.038506] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052104] zbud: loaded [ 3.055586] *** VALIDATE nfs *** [ 3.057092] *** VALIDATE nfs4 *** [ 3.058840] pstore: using deflate compression [ 3.062996] Platform Keyring initialized [ 3.185692] NET: Registered protocol family 38 [ 3.187803] Key type asymmetric registered [ 3.189736] Asymmetric key parser 'x509' registered [ 3.191743] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.195081] io scheduler mq-deadline registered [ 3.196751] io scheduler kyber registered [ 3.198471] io scheduler bfq registered [ 3.200457] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.203545] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.207352] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.210355] ACPI: Power Button [PWRF] [ 3.215710] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.223678] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.237240] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.249166] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.268705] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.297447] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.325631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.330265] Non-volatile memory driver v1.3 [ 3.331845] Linux agpgart interface v0.103 [ 3.368717] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.373328] vda: detected capacity change from 0 to 75087872 [ 3.393769] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397095] vdb: detected capacity change from 0 to 1073741824 [ 3.413894] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416812] vdc: detected capacity change from 0 to 2621440000 [ 3.432988] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.436310] vdd: detected capacity change from 0 to 2621440000 [ 3.454325] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.458089] vde: detected capacity change from 0 to 4294967296 [ 3.476114] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.481607] vdf: detected capacity change from 0 to 4294967296 [ 3.490538] libphy: Fixed MDIO Bus: probed [ 3.497766] usbcore: registered new interface driver usbserial_generic [ 3.501255] usbserial: USB Serial support registered for generic [ 3.504041] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.508852] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.510926] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.514277] mousedev: PS/2 mouse device common for all mice [ 3.518249] rtc_cmos 00:05: RTC can wake from S4 [ 3.521659] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.527110] rtc_cmos 00:05: registered as rtc0 [ 3.529877] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.530334] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.531934] intel_pstate: CPU model not supported [ 3.536153] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.541269] hid: raw HID events driver (C) Jiri Kosina [ 3.544951] usbcore: registered new interface driver usbhid [ 3.547481] usbhid: USB HID core driver [ 3.549122] drop_monitor: Initializing network drop monitor service [ 3.551455] Initializing XFRM netlink socket [ 3.553197] NET: Registered protocol family 10 [ 3.556099] Segment Routing with IPv6 [ 3.557693] NET: Registered protocol family 17 [ 3.560044] mpls_gso: MPLS GSO support [ 3.566892] RAS: Correctable Errors collector initialized. [ 3.569301] AVX version of gcm_enc/dec engaged. [ 3.571603] AES CTR mode by8 optimization enabled [ 3.654347] sched_clock: Marking stable (3654300492, 0)->(4570307799, -916007307) [ 3.658575] registered taskstats version 1 [ 3.660855] Loading compiled-in X.509 certificates [ 3.662973] zswap: loaded using pool lzo/zbud [ 3.688241] Key type big_key registered [ 3.702744] Key type encrypted registered [ 3.704658] ima: No TPM chip found, activating TPM-bypass! [ 3.706864] ima: Allocated hash algorithm: sha1 [ 3.708351] ima: No architecture policies found [ 3.710379] evm: Initialising EVM extended attributes: [ 3.711889] evm: security.selinux [ 3.713187] evm: security.ima [ 3.714416] evm: security.capability [ 3.716221] evm: HMAC attrs: 0x1 [ 3.718503] rtc_cmos 00:05: setting system clock to 2026-09-08 05:37:15 UTC (1788845835) [ 3.724272] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.727297] debug: unmapping init [mem 0xffffffffa5382000-0xffffffffa5658fff] [ 3.740160] Write protecting the kernel read-only data: 28672k [ 3.743509] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.746411] debug: unmapping init [mem 0xffffffffa4314000-0xffffffffa43fffff] [ 3.790333] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.799052] systemd[1]: Detected virtualization kvm. [ 3.801156] systemd[1]: Detected architecture x86-64. [ 3.803319] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.831622] systemd[1]: No hostname configured. [ 3.832843] systemd[1]: Set hostname to . [ 3.834967] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837734] systemd[1]: Initializing machine ID from random generator. [ 3.981082] random: systemd: uninitialized urandom read (16 bytes read) [ 3.984385] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.990687] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993883] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.999101] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.631562] device-mapper: uevent: version 1.0.3 [ 4.635165] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.419788] virtio_net virtio0 ens2: renamed from eth0 [ 5.425803] random: fast init done [ 5.472882] scsi host0: ata_piix [ 5.487618] scsi host1: ata_piix [ 5.489456] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.492310] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.893509] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.222056] random: crng init done [ 10.224292] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.710205] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.952525] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.246298] SELinux: Disabled at runtime. [ 12.311110] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.320443] systemd[1]: Detected virtualization kvm. [ 12.322549] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.912231] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.915826] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.921843] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.925407] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.928650] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.935812] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.942773] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ 13.004450] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.435308] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.746413] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.771052] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.891263] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.903400] EDAC sbridge: Ver: 1.1.2 [ 15.404871] Key type dns_resolver registered [ 15.735342] NFS: Registering the id_resolver key type [ 15.737905] Key type id_resolver registered [ 15.740061] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg631-server login: [ 66.760740] libcfs: loading out-of-tree module taints kernel. [ 66.820805] Key type ._llcrypt registered [ 66.822889] Key type .llcrypt registered [ 66.907684] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 83.338564] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 84.706794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.738064] alg: No test for adler32 (adler32-zlib) [ 86.135733] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 86.819778] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 88.569539] Key type lgssc registered [ 90.405842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.947162] hrtimer: interrupt took 5149892 ns [ 104.783207] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.183588] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 151.703224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.737765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.921101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.939481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.005738] Lustre: lustre-MDT0000: new disk, initializing [ 153.082134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.106685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.495298] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.238066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.315966] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.335575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.339778] Lustre: Skipped 1 previous similar message [ 169.428309] Lustre: lustre-MDT0001: new disk, initializing [ 169.510551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.545871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.552932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.875333] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.611505] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.266632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.522363] Lustre: lustre-OST0000: new disk, initializing [ 185.527962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.532895] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.615481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.267050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.290182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.372268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.325937] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.299562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.476597] Lustre: lustre-OST0001: new disk, initializing [ 207.479856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.489735] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.552428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.810719] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.100182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.118919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.171431] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.531859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.817563] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.243459] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.525262] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing yml_node [ 244.685710] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 246.862274] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 248.967652] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 250.107621] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 01:41:19 EDT 2026 [ 266.435947] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 267.921710] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.693085] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.699706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.703965] Lustre: Skipped 2 previous similar messages [ 286.708353] Lustre: Skipped 1 previous similar message [ 289.029508] Lustre: server umount lustre-MDT0000 complete [ 292.391445] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846124 with bad export cookie 7416194994425825929 [ 292.395970] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.403852] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.930387] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.940204] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.944533] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 296.946126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.960363] Lustre: Skipped 2 previous similar messages [ 296.974950] Lustre: Skipped 1 previous similar message [ 297.954258] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.957042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.986258] Lustre: Skipped 1 previous similar message [ 298.838987] Lustre: server umount lustre-MDT0001 complete [ 302.446919] Lustre: server umount lustre-OST0000 complete [ 306.282386] Lustre: server umount lustre-OST0001 complete [ 313.220338] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 324.059059] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 356.786978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.964188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.984655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.083757] Lustre: lustre-MDT0000: new disk, initializing [ 357.172874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.189281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.743908] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 373.710705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.821541] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.870054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.880508] Lustre: Skipped 1 previous similar message [ 373.939722] Lustre: lustre-MDT0001: new disk, initializing [ 374.044519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.092455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.099498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.579773] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 388.843054] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.152684] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 400.679675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.989596] Lustre: lustre-OST0000: new disk, initializing [ 400.994301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 401.000333] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 401.088806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.823154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 402.840769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.920573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.176742] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 414.834960] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 418.791704] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 418.978509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 423.194063] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 423.373372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 428.527537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.534825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.548337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.558786] Lustre: Skipped 1 previous similar message [ 431.331217] Lustre: server umount lustre-OST0000 complete [ 435.683040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 435.688928] LustreError: Skipped 1 previous similar message [ 435.693184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.703216] Lustre: Skipped 1 previous similar message [ 435.708484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 441.113499] Lustre: server umount lustre-MDT0000 complete [ 443.875620] LustreError: 15073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.895531] LustreError: 15073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 444.611279] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846276 with bad export cookie 7416194994425827462 [ 444.614080] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.622339] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 444.971753] Lustre: server umount lustre-MDT0001 complete [ 450.315625] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:44:39 (1788846279) [ 469.978668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.387331] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.459306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.297601] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 475.629110] LustreError: 19912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.270153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.969192] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 497.886201] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 503.971736] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 506.168783] Lustre: Failing over lustre-MDT0000 [ 506.338231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.339776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.346548] Lustre: Skipped 1 previous similar message [ 506.348253] LustreError: 20963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.348262] LustreError: 20963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 506.403846] Lustre: server umount lustre-MDT0000 complete [ 524.354038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.469290] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.114901] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 529.894939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 529.900591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 529.919879] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 536.590761] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 540.130134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.131192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.137109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.137186] Lustre: Skipped 2 previous similar messages [ 540.147818] Lustre: Skipped 2 previous similar messages [ 546.152802] Lustre: server umount lustre-MDT0000 complete [ 549.619830] LustreError: 21481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846381 with bad export cookie 7416194994425828876 [ 549.635725] LustreError: 21481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 549.638848] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.915510] Lustre: server umount lustre-MDT0001 complete [ 556.949266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.304246] LustreError: 23247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.324560] LustreError: 23247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 557.393588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.397869] Lustre: Skipped 2 previous similar messages [ 561.220149] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 568.885757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.569272] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 580.178293] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.629728] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.657473] Lustre: Failing over lustre-MDT0000 [ 587.745392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.758778] Lustre: Skipped 1 previous similar message [ 587.761130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 587.772663] Lustre: Skipped 4 previous similar messages [ 588.007595] Lustre: server umount lustre-MDT0000 complete [ 592.869797] LustreError: 23248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.880894] LustreError: 23248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 607.115384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.244686] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.566886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 611.415386] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 612.835486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 612.841915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 612.856162] Lustre: Skipped 1 previous similar message [ 612.878821] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.196218] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 628.196499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.197774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.252588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.256679] Lustre: Skipped 1 previous similar message [ 630.058330] Lustre: server umount lustre-MDT0000 complete [ 633.226973] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846465 with bad export cookie 7416194994425829814 [ 633.238094] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.240090] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 633.605781] Lustre: server umount lustre-MDT0001 complete [ 640.991443] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:47:50 (1788846470) [ 655.036974] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 663.267606] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 697.637737] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 707.596558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.811281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 707.833061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 707.924396] Lustre: lustre-MDT0000: new disk, initializing [ 707.980030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.983893] Lustre: Skipped 2 previous similar messages [ 708.005776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 711.628318] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 721.104400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.198066] Lustre: 29620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 721.395289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 725.873498] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 734.871067] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 740.193394] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 746.644635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 746.949592] Lustre: 31587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 748.499774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 748.541888] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 748.593282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 748.613566] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 752.711923] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 761.863767] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 766.836441] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 789.485721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.486777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 789.511387] Lustre: Skipped 1 previous similar message [ 789.512891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.200192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 799.398597] Lustre: server umount lustre-OST0000 complete [ 809.734524] Lustre: server umount lustre-MDT0000 complete [ 809.956343] LustreError: 30494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 809.980573] LustreError: 30494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 813.440739] LustreError: 31586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846645 with bad export cookie 7416194994425830220 [ 813.451089] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 813.451399] LustreError: 31586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 813.965819] Lustre: server umount lustre-MDT0001 complete [ 824.394790] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 827.556958] Key type lgssc unregistered [ 827.961656] LNet: 33558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.983424] LNetError: 33558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.001982] LNet: Removed LNI 192.168.206.131@tcp [ 829.182186] Key type .llcrypt unregistered [ 829.185887] Key type ._llcrypt unregistered [ 856.250551] Key type ._llcrypt registered [ 856.252530] Key type .llcrypt registered [ 856.396313] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 857.898977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.210936] alg: No test for adler32 (adler32-zlib) [ 860.296304] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 860.793505] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 862.592186] Key type lgssc registered [ 864.151018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 872.640297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 872.670192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.196619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.083306] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 883.684351] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.707701] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 888.486960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.810276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 888.980141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.492829] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 901.465397] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.123635] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 913.936687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 914.284972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.442120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 920.331773] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 926.638221] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.747332] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 942.001161] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:52:51 (1788846771) [ 943.407927] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 944.735475] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 949.807665] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:52:59 (1788846779) [ 957.136720] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:53:06 (1788846786) [ 958.110630] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 966.276276] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 966.281179] Lustre: Skipped 2 previous similar messages [ 968.090594] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:53:17 (1788846797) [ 971.524414] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 971.532938] Lustre: Skipped 3 previous similar messages [ 986.635682] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 986.646577] Lustre: Skipped 6 previous similar messages [ 989.232575] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:53:38 (1788846818) [ 991.488733] Lustre: *** cfs_fail_loc=131b, val=2*** [ 994.428909] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1002.582473] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:53:52 (1788846832) [ 1004.232409] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1004.238258] Lustre: Skipped 3 previous similar messages [ 1015.427657] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:54:04 (1788846844) [ 1031.142869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1031.148789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.150248] LustreError: Skipped 1 previous similar message [ 1031.151101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.174185] Lustre: Skipped 1 previous similar message [ 1035.227538] Lustre: server umount lustre-OST0000 complete [ 1041.383402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.389293] Lustre: Skipped 1 previous similar message [ 1041.393488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.402655] Lustre: Skipped 2 previous similar messages [ 1045.591984] Lustre: server umount lustre-MDT0000 complete [ 1046.498894] LustreError: 35991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.515450] LustreError: 35991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1049.143603] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846880 with bad export cookie 671904085034822645 [ 1049.147425] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.152956] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1049.447413] Lustre: server umount lustre-MDT0001 complete [ 1059.720811] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1062.334585] Key type lgssc unregistered [ 1062.558749] LNet: 42820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.574282] LNetError: 42820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.597885] LNet: Removed LNI 192.168.206.131@tcp [ 1063.522659] Key type .llcrypt unregistered [ 1063.526531] Key type ._llcrypt unregistered [ 1079.546052] Key type ._llcrypt registered [ 1079.551061] Key type .llcrypt registered [ 1079.647586] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 1093.609501] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1094.506448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.613298] alg: No test for adler32 (adler32-zlib) [ 1095.671527] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1095.894107] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1097.560161] Key type lgssc registered [ 1098.524640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.404899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.447241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.630651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1129.691101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1129.746616] Lustre: lustre-MDT0000: new disk, initializing [ 1129.802316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.814650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1133.544749] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1142.937645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.070350] Lustre: 45962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1143.145465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1143.153530] Lustre: Skipped 1 previous similar message [ 1143.222768] Lustre: lustre-MDT0001: new disk, initializing [ 1143.332657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.354885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1143.366673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1147.567328] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1156.440554] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.876458] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.794446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.209689] Lustre: lustre-OST0000: new disk, initializing [ 1169.222977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1169.247529] Lustre: 47930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1169.364431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.340555] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1178.716564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1178.731629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1178.872209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1184.310848] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.795474] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1188.985418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1193.457197] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1193.671674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1199.596317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.614721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1199.618158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.627190] LustreError: Skipped 1 previous similar message [ 1199.637835] Lustre: Skipped 1 previous similar message [ 1204.715034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.724077] Lustre: Skipped 1 previous similar message [ 1209.827301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.336695] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1210.523425] Lustre: server umount lustre-OST0000 complete [ 1214.945272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.950524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.953403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.962062] Lustre: Skipped 2 previous similar messages [ 1214.981429] Lustre: Skipped 2 previous similar messages [ 1220.065904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.075477] Lustre: Skipped 1 previous similar message [ 1228.256151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1228.416527] Lustre: server umount lustre-MDT0000 complete [ 1231.490815] LustreError: 46758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847063 with bad export cookie 12089181731791353578 [ 1231.499403] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.508942] LustreError: 46758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1231.800689] Lustre: server umount lustre-MDT0001 complete [ 1240.081920] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:57:49 (1788847069) [ 1246.402111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.729541] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.746157] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1246.794837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.201338] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1251.812645] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.930034] LustreError: 50393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.157400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.437441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.737979] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1267.875532] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1273.441899] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1279.544066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1279.805789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.446570] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1292.800101] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1298.354149] LustreError: 52922:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1299.359384] Lustre: Modifying parameter general.timeout=1 in log params [ 1300.926974] Lustre: Modifying parameter general.timeout=3 in log params [ 1300.933734] Lustre: Skipped 1 previous similar message [ 1303.565868] Lustre: Modifying parameter general.timeout=6 in log params [ 1303.571480] Lustre: Skipped 2 previous similar messages [ 1308.007737] Lustre: Modifying parameter general.timeout=11 in log params [ 1308.010349] Lustre: Skipped 4 previous similar messages [ 1316.391175] Lustre: Modifying parameter general.timeout=19 in log params [ 1316.395149] Lustre: Skipped 7 previous similar messages [ 1332.697876] Lustre: Modifying parameter general.timeout=35 in log params [ 1332.706835] Lustre: Skipped 15 previous similar messages [ 1365.551246] Lustre: Modifying parameter general.timeout=72 in log params [ 1365.554595] Lustre: Skipped 36 previous similar messages [ 1420.730438] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:00:50 (1788847250) [ 1423.209387] Lustre: 59864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1425.508731] Lustre: 59963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1427.653403] Lustre: 60061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1431.090327] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1431.107495] Lustre: Skipped 63 previous similar messages [ 1432.114492] Lustre: 60291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1432.130689] Lustre: 60291:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1462.889988] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1462.919662] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1462.934238] LustreError: 61750:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1467.079621] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:01:36 (1788847296) [ 1477.603560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1477.614641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.632725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.636042] Lustre: Skipped 2 previous similar messages [ 1481.708595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.741718] Lustre: Skipped 1 previous similar message [ 1483.397466] Lustre: server umount lustre-MDT0000 complete [ 1486.489501] LustreError: 50376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847318 with bad export cookie 12089181731791354460 [ 1486.490627] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.502176] LustreError: 50376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1486.684061] Lustre: server umount lustre-MDT0001 complete [ 1500.366443] Lustre: server umount lustre-OST0000 complete [ 1527.278616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.433428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1527.609405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.601185] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1540.163495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.252939] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1540.289436] Lustre: 64282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1544.805648] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1551.551777] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.284756] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1562.067734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1562.206840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1562.420638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.424306] Lustre: Skipped 1 previous similar message [ 1567.790718] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1573.949584] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1575.519472] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1575.525867] Lustre: Skipped 17 previous similar messages [ 1575.575145] Lustre: 66849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1585.639725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1585.643714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.650800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.656100] Lustre: Skipped 3 previous similar messages [ 1588.157827] Lustre: server umount lustre-OST0000 complete [ 1591.777640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1591.786388] LustreError: Skipped 1 previous similar message [ 1596.386912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.398128] Lustre: Skipped 2 previous similar messages [ 1597.766356] Lustre: server umount lustre-MDT0000 complete [ 1601.002551] LustreError: 64275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847432 with bad export cookie 12089181731791357540 [ 1601.004852] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.014315] LustreError: 64275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1601.365521] Lustre: server umount lustre-MDT0001 complete [ 1611.810738] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1614.746709] Key type lgssc unregistered [ 1615.033817] LNet: 68057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.041943] LNetError: 68057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.064864] LNet: Removed LNI 192.168.206.131@tcp [ 1615.984663] Key type .llcrypt unregistered [ 1615.988271] Key type ._llcrypt unregistered [ 1640.371169] Key type ._llcrypt registered [ 1640.375732] Key type .llcrypt registered [ 1640.504715] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1641.452428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.616712] alg: No test for adler32 (adler32-zlib) [ 1642.621961] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1642.855603] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1644.528167] Key type lgssc registered [ 1645.857953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1653.307980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1653.342878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.864986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.556448] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1664.996400] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.018621] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1667.044226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.699833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.893328] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1678.563707] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1683.739717] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1689.896930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.453163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.942662] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1702.247329] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1710.455217] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1720.298555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.314114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.323177] Lustre: Skipped 2 previous similar messages [ 1720.353487] Lustre: Skipped 2 previous similar messages [ 1725.413259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.187365] Lustre: server umount lustre-MDT0000 complete [ 1729.504277] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847561 with bad export cookie 6586385591448057932 [ 1729.513252] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.518871] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1729.894377] Lustre: server umount lustre-MDT0001 complete [ 1744.077333] Lustre: server umount lustre-OST0000 complete [ 1771.450277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.723517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1771.769703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1772.005795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.034435] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1784.293790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.411757] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1784.446437] Lustre: 74338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.920803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1788.911271] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1796.147708] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.775882] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1809.341549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1809.572811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1810.095129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.001629] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1822.610135] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.184376] Lustre: 76912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1826.244896] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1832.417413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.426151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.443377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1832.451293] Lustre: Skipped 2 previous similar messages [ 1832.931426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1838.057648] LustreError: 76639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.070985] LustreError: 76639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1838.198346] Lustre: server umount lustre-OST0000 complete [ 1843.172972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.192094] Lustre: Skipped 1 previous similar message [ 1843.208140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.213489] Lustre: Skipped 2 previous similar messages [ 1848.039614] Lustre: server umount lustre-MDT0000 complete [ 1848.290944] LustreError: 75293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.302524] LustreError: 75293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1851.209461] LustreError: 74329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847682 with bad export cookie 6586385591448058779 [ 1851.212246] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.224150] LustreError: 74329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1851.566244] Lustre: server umount lustre-MDT0001 complete [ 1862.334362] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1864.855682] Key type lgssc unregistered [ 1865.090922] LNet: 78126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.096250] LNetError: 78126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.111619] LNet: Removed LNI 192.168.206.131@tcp [ 1865.932502] Key type .llcrypt unregistered [ 1865.933976] Key type ._llcrypt unregistered [ 1889.646255] Key type ._llcrypt registered [ 1889.649319] Key type .llcrypt registered [ 1889.840155] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1890.994450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.070764] alg: No test for adler32 (adler32-zlib) [ 1892.179382] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1892.439172] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1894.147221] Key type lgssc registered [ 1895.085992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.340413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.350621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.716499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.613883] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1913.831092] LustreError: 79535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.846132] LustreError: 79535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1915.274699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.760506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.117717] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1926.718554] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1932.674476] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1939.167850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.691560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.737225] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1954.446805] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1961.506984] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1962.561834] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1963.715577] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1965.907517] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1965.912107] Lustre: Skipped 1 previous similar message [ 1970.421469] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1970.426542] Lustre: Skipped 3 previous similar messages [ 1978.555153] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1978.560536] Lustre: Skipped 7 previous similar messages [ 1995.028945] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1995.032964] Lustre: Skipped 15 previous similar messages [ 2027.180455] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.184968] Lustre: Skipped 34 previous similar messages [ 2071.688065] Lustre: 87646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2075.156930] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:11:44 (1788847904) [ 2076.755621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2078.277430] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:11:47 (1788847907) [ 2084.326964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2084.339064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.378695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2087.394444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2087.411789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2087.414760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.475495] Lustre: Skipped 1 previous similar message [ 2089.029472] Lustre: server umount lustre-OST0000 complete [ 2095.082457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2095.089280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.108243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.638652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.639788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.669633] Lustre: Skipped 1 previous similar message [ 2102.754121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.767843] Lustre: Skipped 1 previous similar message [ 2107.361326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2107.716883] Lustre: server umount lustre-MDT0000 complete [ 2107.882090] LustreError: 79530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.931846] LustreError: 79530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2113.003611] LustreError: 79536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.055755] LustreError: 79536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2113.305884] LustreError: 79517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847945 with bad export cookie 3877256655003540896 [ 2113.310266] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.318042] LustreError: 79517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2113.717113] Lustre: server umount lustre-MDT0001 complete [ 2128.022097] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2132.554600] Key type lgssc unregistered [ 2132.942096] LNet: 88895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.947536] LNetError: 88895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.963071] LNet: Removed LNI 192.168.206.131@tcp [ 2133.708534] Key type .llcrypt unregistered [ 2133.712671] Key type ._llcrypt unregistered [ 2138.850477] Key type ._llcrypt registered [ 2138.852253] Key type .llcrypt registered [ 2138.978847] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2143.559649] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules [ 2144.487611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.631202] alg: No test for adler32 (adler32-zlib) [ 2145.677303] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2145.969217] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2147.720155] Key type lgssc registered [ 2149.062516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.119960] LustreError: 90093:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2169.442302] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2172.254267] LustreError: 90093:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2179.877343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2179.895261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.337650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.702846] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2191.841754] LustreError: 91089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.864969] LustreError: 91089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2195.126405] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:13:44 (1788848024) [ 2196.964098] LustreError: 91088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.677166] LustreError: 91120:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2199.685458] LustreError: 91120:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2199.695948] LustreError: 91120:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2200.097498] Lustre: server umount lustre-MDT0000 complete [ 2215.111940] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2217.866800] Key type lgssc unregistered [ 2218.132482] LNet: 92615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.137400] LNetError: 92615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.182745] LNet: Removed LNI 192.168.206.131@tcp [ 2220.136258] Key type .llcrypt unregistered [ 2220.140694] Key type ._llcrypt unregistered [ 2237.638308] Key type ._llcrypt registered [ 2237.641167] Key type .llcrypt registered [ 2237.774781] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2239.096590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.112930] alg: No test for adler32 (adler32-zlib) [ 2240.126524] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2240.473097] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2242.176153] Key type lgssc registered [ 2243.474529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.054580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.077078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.903562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.106701] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2264.042560] LustreError: 93829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.094362] LustreError: 93829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2268.612654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.532922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.684211] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2284.473494] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.192756] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.422483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.403428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.688838] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2314.653993] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.886805] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:16:47 (1788848207) [ 2379.932747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2382.358637] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:16:51 (1788848211) [ 2386.917464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.933057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.963607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.501050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.510583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.525553] Lustre: Skipped 1 previous similar message [ 2390.546657] Lustre: Skipped 2 previous similar messages [ 2393.120359] Lustre: server umount lustre-MDT0000 complete [ 2395.619130] LustreError: 93830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.653023] LustreError: 93830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2398.165516] LustreError: 93810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848229 with bad export cookie 13659184563134561952 [ 2398.183593] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.975298] Lustre: server umount lustre-MDT0001 complete [ 2415.070135] Lustre: server umount lustre-OST0000 complete [ 2430.418699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.052382] LustreError: 98134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.287292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.577734] LustreError: 98135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.995086] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2440.688290] LustreError: 98134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.104719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.967768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2454.412259] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2462.520032] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.275917] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.985229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.335737] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2481.356143] LustreError: 98124:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2481.397604] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2481.413279] LustreError: 100217:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2481.424757] LustreError: 100217:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2481.433048] LustreError: 100217:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2481.550455] Lustre: server umount lustre-OST0000 complete [ 2481.561978] LustreError: 100217:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2490.151822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.308166] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2490.318488] LustreError: 98124:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2490.330381] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2490.339066] LustreError: 100680:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2490.343634] LustreError: 100680:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2490.347637] LustreError: 100680:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2490.398069] Lustre: server umount lustre-OST0000 complete [ 2490.400563] LustreError: 100680:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2500.521532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.779278] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2500.786601] Lustre: Found index 0 for lustre-OST0000, updating log [ 2500.799366] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2500.872445] Lustre: lustre-OST0000: new disk, initializing [ 2500.875798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2500.880292] Lustre: 101215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2501.144408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.375866] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2511.279311] Lustre: Failing over lustre-OST0000 [ 2511.335052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.335973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.372773] LustreError: 101200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.385404] LustreError: 101200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2511.545913] Lustre: server umount lustre-OST0000 complete [ 2516.450219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.451251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.466799] Lustre: Skipped 1 previous similar message [ 2516.479724] Lustre: Skipped 1 previous similar message [ 2518.499208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2518.513600] Lustre: Skipped 1 previous similar message [ 2521.679595] Lustre: server umount lustre-MDT0000 complete [ 2525.488138] LustreError: 98115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848357 with bad export cookie 13659184563134564696 [ 2525.489588] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2525.496258] LustreError: 98115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2525.951932] Lustre: server umount lustre-MDT0001 complete [ 2535.326258] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:19:24 (1788848364) [ 2551.198437] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2561.420217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.951085] LustreError: 103559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.969780] LustreError: 103559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2562.093894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.699575] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2574.536911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.609231] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2582.291793] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2588.534424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.157845] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2595.205908] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2599.401211] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2599.418035] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2599.446121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2603.031275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.147739] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2603.281169] Lustre: lustre-OST0001: new disk, initializing [ 2603.285306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2603.295704] Lustre: 105880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2603.494491] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2603.520602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2603.529042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2603.544310] Lustre: Skipped 2 previous similar messages [ 2606.636185] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2606.652682] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2606.652816] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2606.680729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2609.563528] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2612.839387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2612.850669] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2612.958926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2616.784825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2622.254904] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2626.539440] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.550586] LustreError: Skipped 1 previous similar message [ 2626.556299] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.571035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2637.673102] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.31@tcp (stopping) [ 2637.683401] Lustre: Skipped 8 previous similar messages [ 2640.352843] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2640.634581] Lustre: server umount lustre-MDT0001 complete [ 2642.793920] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.817774] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2649.238573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.312853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2649.324071] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2649.374883] Lustre: 103548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2650.616916] LustreError: 107431:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2650.627037] LustreError: 107431:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2650.640580] LustreError: 103551:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2650.719748] LustreError: 107451:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -17 [ 2650.725485] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2650.725485] [ 2653.578255] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2655.220166] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2662.054437] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:21:31 (1788848491) [ 2679.127607] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2699.535094] Lustre: 109493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2729.892827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2734.539916] Lustre: 110630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2823.649611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2823.661630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.677648] Lustre: Skipped 2 previous similar messages [ 2823.689676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.700532] Lustre: Skipped 3 previous similar messages [ 2826.412687] Lustre: server umount lustre-MDT0000 complete [ 2829.289709] LustreError: 103556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.314931] LustreError: 103556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2830.432176] LustreError: 104703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848662 with bad export cookie 13659184563134565627 [ 2830.434647] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.443639] LustreError: 104703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2830.932232] Lustre: server umount lustre-MDT0001 complete [ 2846.699867] Lustre: server umount lustre-OST0000 complete [ 2860.888216] Lustre: server umount lustre-OST0001 complete [ 2866.225709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2877.166053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2886.249085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2897.064611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2917.830835] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2928.625677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.650887] Lustre: lustre-MDT0000: reset Object Index mappings [ 2929.221524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.232120] Lustre: Skipped 1 previous similar message [ 2933.732448] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2942.505628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.530447] Lustre: lustre-MDT0001: reset Object Index mappings [ 2947.868132] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2950.788551] Lustre: 115729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2957.284957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.520283] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2968.232424] LustreError: 116083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.248584] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2968.269034] LustreError: 116083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2973.157684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.365873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 2978.812613] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2978.819100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2979.875542] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2987.856237] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2991.368134] Lustre: 117603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3038.316385] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:27:47 (1788848867) [ 3065.825886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.842943] Lustre: Skipped 3 previous similar messages [ 3065.845702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.861809] Lustre: Skipped 7 previous similar messages [ 3070.834225] Lustre: server umount lustre-MDT0000 complete [ 3074.158393] LustreError: 114562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848905 with bad export cookie 13659184563134612688 [ 3074.163331] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.179296] LustreError: 114562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3080.740381] Lustre: server umount lustre-MDT0001 complete [ 3090.928671] Lustre: server umount lustre-OST0000 complete [ 3100.913288] Lustre: server umount lustre-OST0001 complete [ 3108.466485] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 3119.513652] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3153.551288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.856264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3153.881842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3153.942707] Lustre: lustre-MDT0000: new disk, initializing [ 3154.011958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.018019] Lustre: Skipped 3 previous similar messages [ 3154.040915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3157.789749] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3164.214640] Lustre: server umount lustre-MDT0000 complete [ 3173.167162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.431702] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3173.446121] Lustre: 121873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3173.462847] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3179.607475] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3184.774465] Lustre: server umount lustre-MDT0000 complete [ 3194.221576] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:30:23 (1788849023) [ 3195.829602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3197.603503] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:30:26 (1788849026) [ 3215.766029] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3226.255779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.319763] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3239.906857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.010661] Lustre: 123926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3240.078585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3240.083532] Lustre: Skipped 1 previous similar message [ 3240.173873] Lustre: lustre-MDT0001: new disk, initializing [ 3240.357585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3240.375406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3244.951876] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3249.962357] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3257.127391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.433327] Lustre: lustre-OST0000: new disk, initializing [ 3257.437078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3257.451450] Lustre: 125503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3259.300569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3259.313983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3259.433628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3263.924539] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3274.022857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.176603] Lustre: lustre-OST0001: new disk, initializing [ 3274.179472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3274.184318] Lustre: 126374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3275.940217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3275.950769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3276.061785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3280.105984] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3290.238259] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3293.788976] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3608.634646] Lustre: 132777:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3608.643331] Lustre: 132777:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3608.648672] Lustre: 132777:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3608.654228] Lustre: 132777:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3608.658058] Lustre: 132777:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3608.662063] Lustre: 132777:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3836.335465] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:41:05 (1788849665) [ 3850.211561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3850.222585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.234421] Lustre: Skipped 2 previous similar messages [ 3850.240954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.245274] Lustre: Skipped 4 previous similar messages [ 3854.824971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.837349] Lustre: Skipped 1 previous similar message [ 3856.407541] Lustre: server umount lustre-MDT0000 complete [ 3859.105663] LustreError: 124664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849690 with bad export cookie 13659184563134678985 [ 3859.114097] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.114507] LustreError: 124664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3859.937371] LustreError: 123932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.940480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3859.947966] LustreError: 123932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3859.962146] Lustre: Skipped 5 previous similar messages [ 3865.462851] Lustre: server umount lustre-MDT0001 complete [ 3874.152953] Lustre: server umount lustre-OST0000 complete [ 3882.816673] Lustre: server umount lustre-OST0001 complete [ 3887.106164] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 3894.444832] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3919.332599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.519614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3919.541961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3919.606514] Lustre: lustre-MDT0000: new disk, initializing [ 3919.674396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.681082] Lustre: Skipped 5 previous similar messages [ 3919.690040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3922.783613] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3930.910636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.022201] Lustre: 135824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.089214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3931.092907] Lustre: Skipped 1 previous similar message [ 3931.153738] Lustre: lustre-MDT0001: new disk, initializing [ 3931.257620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3931.268694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3934.862958] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3941.692863] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3945.920289] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3950.722194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3950.939925] Lustre: lustre-OST0000: new disk, initializing [ 3950.944237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3950.951246] Lustre: 137793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3952.227961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3952.235506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3952.340377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3955.889389] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3963.503756] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3969.907519] Lustre: lustre-MDD0000: changelog on [ 3971.641152] Lustre: lustre-MDD0001: changelog on [ 5056.859431] Lustre: lustre-MDD0001: changelog off [ 5057.695704] Lustre: 135831:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5058.078379] Lustre: lustre-MDD0000: changelog off [ 5063.136923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5063.141370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.147775] Lustre: Skipped 3 previous similar messages [ 5063.150050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5063.154340] Lustre: Skipped 2 previous similar messages [ 5063.653143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5066.569493] Lustre: server umount lustre-OST0000 complete [ 5068.770170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.780888] Lustre: Skipped 1 previous similar message [ 5068.784559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5068.787910] Lustre: Skipped 2 previous similar messages [ 5073.892464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.901996] Lustre: Skipped 1 previous similar message [ 5074.725696] Lustre: server umount lustre-MDT0000 complete [ 5076.856947] LustreError: 135814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850908 with bad export cookie 13659184563135496977 [ 5076.860289] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5076.864537] LustreError: 135814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5077.043521] Lustre: server umount lustre-MDT0001 complete [ 5082.968934] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5084.678345] Key type lgssc unregistered [ 5084.864626] LNet: 146809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5084.870670] LNetError: 146809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5084.883653] LNet: Removed LNI 192.168.206.131@tcp [ 5085.372232] Key type .llcrypt unregistered [ 5085.375752] Key type ._llcrypt unregistered [ 5094.874996] Key type ._llcrypt registered [ 5094.876392] Key type .llcrypt registered [ 5094.928304] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5102.698326] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5103.194679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5103.218966] alg: No test for adler32 (adler32-zlib) [ 5104.111257] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5104.252754] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5105.856340] Key type lgssc registered [ 5106.368453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5121.177420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5121.193832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.298557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5122.313062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5122.352399] Lustre: lustre-MDT0000: new disk, initializing [ 5122.397270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.407915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5124.187745] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5129.367660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.421048] Lustre: 149980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5129.451254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5129.453511] Lustre: Skipped 1 previous similar message [ 5129.496135] Lustre: lustre-MDT0001: new disk, initializing [ 5129.568821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5129.588435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5129.593145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.290414] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5136.112713] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5138.709729] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5142.090381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5142.242095] Lustre: lustre-OST0000: new disk, initializing [ 5142.245550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5142.250150] Lustre: 151947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5142.287981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.094037] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5149.738769] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5151.756567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5151.763887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5151.818279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5158.830616] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_module obdecho/obdecho [ 5159.773484] LustreError: 152949:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5159.777325] LustreError: 152949:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5160.577711] LustreError: 153045:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5167.075734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.077600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5167.086070] Lustre: Skipped 1 previous similar message [ 5172.054458] Lustre: server umount lustre-MDT0000 complete [ 5173.857780] LustreError: 149973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851005 with bad export cookie 15401019730698419564 [ 5173.860540] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5173.867261] LustreError: 149973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5174.003192] Lustre: server umount lustre-MDT0001 complete [ 5186.220816] Lustre: server umount lustre-OST0000 complete [ 5195.860648] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5200.539193] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5216.028814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.131960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5216.148769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5216.193889] Lustre: lustre-MDT0000: new disk, initializing [ 5216.227666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.240051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5217.973641] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5222.635839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.684280] Lustre: 156752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5222.717907] LustreError: 156763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.721242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5222.723770] LustreError: 156763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5222.726243] Lustre: Skipped 1 previous similar message [ 5222.767283] Lustre: lustre-MDT0001: new disk, initializing [ 5222.806068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5222.820693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5222.824755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5224.440706] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5228.555900] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.654906] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5233.229888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.346965] Lustre: lustre-OST0000: new disk, initializing [ 5233.349525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5233.353062] Lustre: 158722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.392366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5235.534874] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5237.440106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5237.444141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5237.484218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5239.526310] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5241.150087] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5241.239146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5242.859671] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5242.940968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5247.971082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5247.973528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.975757] LustreError: Skipped 1 previous similar message [ 5247.976403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5247.981115] Lustre: Skipped 2 previous similar messages [ 5247.986650] Lustre: Skipped 3 previous similar messages [ 5249.751224] Lustre: server umount lustre-OST0000 complete [ 5253.091146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.093891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5253.097556] Lustre: Skipped 1 previous similar message [ 5253.102634] Lustre: Skipped 1 previous similar message [ 5257.369371] Lustre: server umount lustre-MDT0000 complete [ 5258.212426] LustreError: 159302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.219759] LustreError: 159302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5258.798615] LustreError: 156745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851090 with bad export cookie 15401019730698420509 [ 5258.802638] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.804491] LustreError: 156745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5258.932282] Lustre: server umount lustre-MDT0001 complete [ 5260.440257] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:04:50 (1788851090) [ 5261.121952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5261.880616] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:04:52 (1788851092) [ 5264.913657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.112191] LustreError: 161130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.149978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.784888] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5270.143273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.766947] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5274.379725] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5276.379248] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5279.029465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5281.482447] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5284.273711] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5291.991337] LustreError: 163707:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x3ade9a31 (000000003d6236cc) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5294.563297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.564981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5294.568496] Lustre: Skipped 1 previous similar message [ 5298.145865] Lustre: server umount lustre-MDT0000 complete [ 5299.410300] LustreError: 161112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851131 with bad export cookie 15401019730698421391 [ 5299.412814] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5299.414623] LustreError: 161112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5299.681446] LustreError: 161126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5299.681599] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.693826] Lustre: Skipped 1 previous similar message [ 5299.695828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5299.698054] Lustre: Skipped 3 previous similar messages [ 5304.801709] LustreError: 161126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.808916] LustreError: 161126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5305.591613] Lustre: server umount lustre-MDT0001 complete [ 5307.875877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5312.312915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.391097] LustreError: 164967:0:(llog.c:1655:llog_backup()) MGC192.168.206.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5312.393520] Lustre: 164967:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5329.376871] LustreError: 164972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.383444] LustreError: 164972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5346.149573] LustreError: 164973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.158742] LustreError: 164973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5373.024358] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5373.028143] Lustre: 164967:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5379.552786] LustreError: 164990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.557989] LustreError: 164990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 5413.867507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5413.869817] Lustre: Skipped 2 previous similar messages [ 5415.438388] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5418.820740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.982044] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5419.872285] LustreError: 165710:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5419.876633] LustreError: 165710:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000d5ad5378 x1875746159165056/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788851251 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5419.884771] LustreError: 165710:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5420.000744] LustreError: 148206:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008a580270 x1875746159169152/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5421.874870] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5424.589610] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.130247] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5425.159951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5427.572120] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5428.210325] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5439.066465] Lustre: server umount lustre-MDT0000 complete [ 5440.485088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5440.485195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.496808] Lustre: Skipped 1 previous similar message [ 5440.790131] LustreError: 161111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851272 with bad export cookie 15401019730698427831 [ 5440.790508] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.798327] LustreError: 161111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5441.042370] Lustre: server umount lustre-MDT0001 complete [ 5452.969506] Lustre: server umount lustre-OST0000 complete [ 5461.055821] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5464.635144] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5477.696455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.804459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5477.815839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5477.853317] Lustre: lustre-MDT0000: new disk, initializing [ 5477.882271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.884621] Lustre: Skipped 1 previous similar message [ 5477.891266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5479.380272] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5483.773855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.814857] Lustre: 170036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5483.916379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5485.371098] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5489.307327] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5491.416936] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5494.021670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5494.135219] Lustre: 172008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5496.316891] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5500.920847] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5500.935996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5500.996767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5502.985173] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5503.082451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5504.970915] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5505.073982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5506.018822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5506.020254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.023017] LustreError: Skipped 1 previous similar message [ 5506.028270] Lustre: Skipped 1 previous similar message [ 5506.030969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5506.034913] Lustre: Skipped 4 previous similar messages [ 5507.890616] Lustre: server umount lustre-OST0000 complete [ 5509.602602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5515.751607] Lustre: server umount lustre-MDT0000 complete [ 5516.256933] LustreError: 170048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.263089] LustreError: 170048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 5517.308213] LustreError: 170030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851349 with bad export cookie 15401019730698444631 [ 5517.311458] LustreError: 170030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5517.311472] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5517.445525] Lustre: server umount lustre-MDT0001 complete [ 5520.789738] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:09:11 (1788851351) [ 5523.296088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.186295] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5528.901301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.812287] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5533.742686] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5536.028727] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5539.002495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5541.609969] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5544.664487] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5558.243411] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:09:48 (1788851388) [ 5563.877304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5563.877809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.882278] LustreError: Skipped 1 previous similar message [ 5563.882978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.887405] Lustre: Skipped 3 previous similar messages [ 5563.889529] Lustre: Skipped 3 previous similar messages [ 5565.469571] Lustre: server umount lustre-OST0000 complete [ 5581.280120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5581.344930] Lustre: server umount lustre-MDT0000 complete [ 5582.764230] LustreError: 174396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851414 with bad export cookie 15401019730698445499 [ 5582.768101] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.770086] LustreError: 174396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5582.897555] Lustre: server umount lustre-MDT0001 complete [ 5586.759199] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5587.844584] Key type lgssc unregistered [ 5587.978503] LNet: 178161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5587.982565] LNetError: 178161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5587.990415] LNet: Removed LNI 192.168.206.131@tcp [ 5588.368468] Key type .llcrypt unregistered [ 5588.369896] Key type ._llcrypt unregistered [ 5596.730452] Key type ._llcrypt registered [ 5596.732181] Key type .llcrypt registered [ 5596.771782] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5597.133167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5597.145848] alg: No test for adler32 (adler32-zlib) [ 5598.015136] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5598.112276] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5599.696134] Key type lgssc registered [ 5600.084784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5603.309657] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5603.323308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.784510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5623.904234] LustreError: 179377:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 5629.024641] LustreError: 179377:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5629.031039] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5629.033570] LustreError: MGC192.168.206.131@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5629.038187] LustreError: 179377:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5629.041157] LustreError: 179377:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 5629.044517] LustreError: 179377:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5629.079077] Lustre: server umount lustre-OST0000 complete [ 5629.080960] LustreError: 179377:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5632.619095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.813107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.439445] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5638.694071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.827718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5640.539866] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5643.752099] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.098712] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5648.834516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5648.976794] LustreError: 181689:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5648.985753] Lustre: 181689:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.131@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5649.058781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.654893] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5654.601875] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5655.430353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.433125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.434217] Lustre: Skipped 2 previous similar messages [ 5655.436434] Lustre: Skipped 1 previous similar message [ 5660.642379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.644376] Lustre: Skipped 2 previous similar messages [ 5661.589771] Lustre: server umount lustre-MDT0000 complete [ 5663.007177] LustreError: 180608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851494 with bad export cookie 1185408531075945930 [ 5663.010208] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.011575] LustreError: 180608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5663.135776] Lustre: server umount lustre-MDT0001 complete [ 5674.715564] Lustre: server umount lustre-OST0000 complete [ 5678.137357] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5681.643672] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5694.620501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.698976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5694.711038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5694.745133] Lustre: lustre-MDT0000: new disk, initializing [ 5694.765300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.771068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5696.255662] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5700.921721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.973334] Lustre: 185206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5701.006475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5701.008954] Lustre: Skipped 1 previous similar message [ 5701.049851] Lustre: lustre-MDT0001: new disk, initializing [ 5701.100818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.121432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5701.127996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5702.846079] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5706.906720] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5709.117729] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5712.523894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.675780] Lustre: lustre-OST0000: new disk, initializing [ 5712.678823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5712.683451] Lustre: 187173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5712.744372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.825175] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5720.861879] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5722.128748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5722.135615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5722.188703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5722.818617] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5722.904118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5724.675525] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5724.774707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5727.201995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.204703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.209632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.212214] LustreError: Skipped 1 previous similar message [ 5727.218635] Lustre: Skipped 1 previous similar message [ 5731.673283] Lustre: server umount lustre-OST0000 complete [ 5736.929477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5736.934553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.940792] Lustre: Skipped 1 previous similar message [ 5736.943935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5739.452456] Lustre: server umount lustre-MDT0000 complete [ 5741.060445] LustreError: 185199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851572 with bad export cookie 1185408531075946630 [ 5741.064652] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5741.065830] LustreError: 185199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5741.221882] Lustre: server umount lustre-MDT0001 complete [ 5744.935143] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:12:55 (1788851575) [ 5752.539373] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5753.822615] Key type lgssc unregistered [ 5753.969403] LNet: 190111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5753.972617] LNetError: 190111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5753.980402] LNet: Removed LNI 192.168.206.131@tcp [ 5754.358672] Key type .llcrypt unregistered [ 5754.360554] Key type ._llcrypt unregistered [ 5762.563913] Key type ._llcrypt registered [ 5762.565173] Key type .llcrypt registered [ 5762.613264] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5763.068739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5763.077768] alg: No test for adler32 (adler32-zlib) [ 5763.941406] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5764.034381] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5765.616131] Key type lgssc registered [ 5765.993935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5768.675979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5768.682484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.849685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.317330] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5774.619930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.805810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5777.167670] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5780.792450] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5783.142093] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5786.710353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5786.812254] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5786.814394] LustreError: 193165:0:(llog.c:1696:llog_backup()) MGC192.168.206.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5786.818561] Lustre: 193165:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.131@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5786.875619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.646500] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5792.586239] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5795.297053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5795.301068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.306120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.346346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.346783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.394147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5799.396835] Lustre: Skipped 4 previous similar messages [ 5799.529358] Lustre: server umount lustre-MDT0000 complete [ 5800.864449] LustreError: 191283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851632 with bad export cookie 2306212174191647566 [ 5800.866972] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.870259] LustreError: 191283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5801.424328] LustreError: 192029:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006fe25d9f x1875746850098944/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5801.476363] Lustre: server umount lustre-MDT0001 complete [ 5813.149137] Lustre: server umount lustre-OST0000 complete [ 5816.609070] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5820.336406] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5833.599824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.694878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5833.708820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5833.751127] Lustre: lustre-MDT0000: new disk, initializing [ 5833.775484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.782361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5835.165442] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5839.445352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.491642] Lustre: 196686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5839.508907] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5839.510579] Lustre: Skipped 1 previous similar message [ 5839.511820] LustreError: 197467:0:(llog.c:1696:llog_backup()) MGC192.168.206.131@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5839.514402] LustreError: 197467:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5839.516459] Lustre: 197467:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5839.524404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5839.526736] Lustre: Skipped 1 previous similar message [ 5839.559742] Lustre: lustre-MDT0001: new disk, initializing [ 5839.593511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.607452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5839.611245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5841.044613] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5844.839063] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5847.153402] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5849.769617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5849.883917] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5849.885821] Lustre: Skipped 1 previous similar message [ 5849.888541] LustreError: 198628:0:(llog.c:1696:llog_backup()) MGC192.168.206.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5849.895023] LustreError: 198628:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5849.898287] Lustre: 198628:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5849.902919] Lustre: 198628:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5849.912603] Lustre: lustre-OST0000: new disk, initializing [ 5849.915776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5849.920115] Lustre: 198652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5849.959268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.656413] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5854.214454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5854.219031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5854.263371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5857.204945] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5859.017503] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5859.113374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5860.842937] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5860.942942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5864.421564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5864.421648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.425551] LustreError: Skipped 1 previous similar message [ 5864.426562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.431647] Lustre: Skipped 2 previous similar messages [ 5869.539368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.542352] Lustre: Skipped 2 previous similar messages [ 5876.192452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5876.247151] Lustre: server umount lustre-OST0000 complete [ 5879.777671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.778508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.782682] Lustre: Skipped 1 previous similar message [ 5879.785396] Lustre: Skipped 2 previous similar messages [ 5883.736191] Lustre: server umount lustre-MDT0000 complete [ 5884.896921] LustreError: 198264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.901754] LustreError: 198264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5884.939066] LustreError: 196722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851716 with bad export cookie 2306212174191648280 [ 5884.941614] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.944238] LustreError: 196722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5885.050132] Lustre: server umount lustre-MDT0001 complete [ 5888.081996] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:15:18 (1788851718) [ 5893.612988] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5897.162067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.278904] LustreError: 201773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.298748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5898.847057] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5902.305241] LustreError: 201774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.397068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.662974] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5906.113094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5909.380503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.026471] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5915.628216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5915.716735] Lustre: lustre-OST0001: new disk, initializing [ 5915.718881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5915.721386] Lustre: 204079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5918.351983] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5920.314824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5920.321079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5920.365352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5924.761912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5930.350995] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5931.515424] Lustre: DEBUG MARKER: ADD OST3 [ 5934.203458] loop: module loaded [ 5934.235585] loop0: detected capacity change from 0 to 204800000 [ 5934.335705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5934.342821] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5934.351785] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5938.500283] loop0: detected capacity change from 0 to 204800000 [ 5940.283744] Lustre: DEBUG MARKER: STOP OST3 [ 5942.354405] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5942.507566] Lustre: lustre-OST0003: new disk, initializing [ 5942.511868] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5942.515781] Lustre: 206009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5942.586258] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5942.590595] Lustre: Skipped 3 previous similar messages [ 5943.720143] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5943.780703] Lustre: Failing over lustre-OST0003 [ 5943.783540] LustreError: lustre-OST0003-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -19 [ 5943.793472] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.799993] LustreError: 206200:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0003-osc-MDT0000: cannot cleanup orphans: rc = -19 [ 5943.800555] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 5943.883871] Lustre: server umount lustre-OST0003 complete [ 5945.760254] LustreError: 206193:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5946.471769] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5947.699301] LustreError: 203259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.674791] Lustre: DEBUG MARKER: START OST3 again [ 5953.900744] LustreError: 203260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.922338] LustreError: 203260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5956.496084] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.630798] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5957.929296] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5958.598809] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5958.600292] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5958.618651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5958.634681] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5958.666825] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5959.763538] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5974.527274] Lustre: Failing over lustre-OST0003 [ 5974.576073] Lustre: server umount lustre-OST0003 complete [ 5977.062025] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.064029] LustreError: 203259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.067422] Lustre: Skipped 2 previous similar messages [ 5977.073776] LustreError: 203259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5989.856913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.860797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.213567] Lustre: server umount lustre-MDT0000 complete [ 5995.735584] LustreError: 204078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851827 with bad export cookie 2306212174191649162 [ 5995.742750] LustreError: 204078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5995.745168] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.537234] LustreError: 201768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.537576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.544984] LustreError: 201768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5997.550323] Lustre: Skipped 5 previous similar messages [ 6010.336170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6010.460983] Lustre: server umount lustre-MDT0001 complete [ 6012.053573] Lustre: server umount lustre-OST0000 complete [ 6013.468141] Lustre: server umount lustre-OST0001 complete [ 6015.789501] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 6018.894597] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6033.605462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.698088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6033.711580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6033.754926] Lustre: lustre-MDT0000: new disk, initializing [ 6033.788846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.791897] Lustre: Skipped 1 previous similar message [ 6033.798205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6035.305073] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6040.559762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.624844] Lustre: 211543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6040.745854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.591507] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6047.140994] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6049.300920] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.295107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.427409] Lustre: lustre-OST0000: new disk, initializing [ 6052.429685] Lustre: Skipped 1 previous similar message [ 6052.432363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6052.435514] Lustre: Skipped 2 previous similar messages [ 6052.438659] Lustre: 213509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6054.469761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6054.473567] Lustre: Skipped 1 previous similar message [ 6054.476195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6054.519716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6054.931099] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6059.299288] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6060.880802] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6060.950706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6062.404785] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6062.489541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6065.122091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.123325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.126466] LustreError: Skipped 1 previous similar message [ 6065.127230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.132972] Lustre: Skipped 1 previous similar message [ 6065.140767] Lustre: Skipped 11 previous similar messages [ 6069.280346] Lustre: server umount lustre-OST0000 complete [ 6076.696139] Lustre: server umount lustre-MDT0000 complete [ 6077.953981] LustreError: 213508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851909 with bad export cookie 2306212174191651437 [ 6077.957419] LustreError: 213508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6077.957492] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.072546] Lustre: server umount lustre-MDT0001 complete [ 6079.176078] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:18:29 (1788851909) [ 6084.303793] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 6087.582043] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6107.559281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.665294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6107.686413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6107.742696] Lustre: lustre-MDT0000: new disk, initializing [ 6107.799914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6110.470782] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6116.411290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.481116] Lustre: 218109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6116.628744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6119.278190] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6124.728145] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6127.389219] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6130.753624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6130.903740] Lustre: 220076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6132.089495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6133.385043] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6137.364065] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6139.099863] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6139.184282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6141.388577] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6141.480980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6147.557041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.557321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.566851] Lustre: Skipped 2 previous similar messages [ 6147.572038] LustreError: Skipped 1 previous similar message [ 6148.602786] Lustre: server umount lustre-OST0000 complete [ 6156.746837] Lustre: server umount lustre-MDT0000 complete [ 6157.794837] LustreError: 218121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6157.802533] LustreError: 218121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6158.394890] LustreError: 218104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851990 with bad export cookie 2306212174191652326 [ 6158.397433] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6158.400786] LustreError: 218104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6158.550842] Lustre: server umount lustre-MDT0001 complete [ 6162.368569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.642624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.647094] Lustre: Skipped 5 previous similar messages [ 6164.703248] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6169.598858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.944461] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6175.821641] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6178.752755] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6182.231386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.136573] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6188.267319] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6202.850502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6202.854506] LustreError: Skipped 2 previous similar messages [ 6202.858551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.860962] Lustre: Skipped 9 previous similar messages [ 6215.136134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6215.218739] Lustre: server umount lustre-OST0000 complete [ 6223.206369] Lustre: server umount lustre-MDT0000 complete [ 6224.644296] LustreError: 222273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852056 with bad export cookie 2306212174191653208 [ 6224.647789] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6224.648636] LustreError: 222273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6224.760264] Lustre: server umount lustre-MDT0001 complete [ 6323.293695] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 6324.961278] Key type lgssc unregistered [ 6325.161719] LNet: 225969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6325.168677] LNetError: 225969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6325.178930] LNet: Removed LNI 192.168.206.131@tcp [ 6325.673279] Key type .llcrypt unregistered [ 6325.675564] Key type ._llcrypt unregistered [ 6339.052369] Key type ._llcrypt registered [ 6339.054139] Key type .llcrypt registered [ 6339.109292] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 6346.353547] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6346.696615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6346.705184] alg: No test for adler32 (adler32-zlib) [ 6347.571939] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 6347.664064] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 6349.256175] Key type lgssc registered [ 6349.640350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6363.987989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.001939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.101381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6365.115439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6365.155769] Lustre: lustre-MDT0000: new disk, initializing [ 6365.198425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.209554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6367.282291] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6373.702559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.778497] Lustre: 229301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6373.849329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6373.852792] Lustre: Skipped 1 previous similar message [ 6373.926687] Lustre: lustre-MDT0001: new disk, initializing [ 6373.991186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6374.031592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6374.037833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6377.260462] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6383.678724] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6387.031552] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6391.205539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.363515] Lustre: lustre-OST0000: new disk, initializing [ 6391.368682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6391.375967] Lustre: 231272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6391.439800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.159982] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6400.147347] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6401.545822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6401.552704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6401.597505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6402.272851] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6402.368595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6404.337500] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6404.435867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6406.630870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.630964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.644217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.649341] LustreError: Skipped 1 previous similar message [ 6406.654574] Lustre: Skipped 1 previous similar message [ 6407.331168] Lustre: server umount lustre-OST0000 complete [ 6409.696892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6409.700840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.707591] Lustre: Skipped 1 previous similar message [ 6409.710339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6411.746749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.747417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6411.759871] Lustre: Skipped 1 previous similar message [ 6415.009454] Lustre: server umount lustre-MDT0000 complete [ 6416.727855] LustreError: 229340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852248 with bad export cookie 1997461585960709933 [ 6416.732139] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.738846] LustreError: 229340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6416.895466] Lustre: server umount lustre-MDT0001 complete [ 6425.599454] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6430.998199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.195244] LustreError: 234200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.204797] LustreError: 234200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6431.222374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.596240] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6436.322671] LustreError: 234201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.083150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6439.266341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6441.534964] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6443.102367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6446.237290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.305764] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6453.756696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.847231] Lustre: lustre-OST0001: new disk, initializing [ 6453.850166] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6453.854485] Lustre: 236505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6453.904863] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6453.908118] Lustre: Skipped 1 previous similar message [ 6456.659657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6456.666100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6456.720740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6456.820538] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6463.103294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6467.519397] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6473.144966] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:25:03 (1788852303) [ 6475.234437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6475.239059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.247218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.206045] Lustre: server umount lustre-MDT0000 complete [ 6477.282834] LustreError: 234201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.290174] LustreError: 234201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6477.828367] LustreError: 234181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852309 with bad export cookie 1997461585960710815 [ 6477.831768] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6477.835673] LustreError: 234181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6478.003299] Lustre: server umount lustre-MDT0001 complete [ 6489.618538] Lustre: server umount lustre-OST0000 complete [ 6501.621453] Lustre: server umount lustre-OST0001 complete [ 6504.367308] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 6508.838183] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6525.463313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.575824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6525.592167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6525.637236] Lustre: lustre-MDT0000: new disk, initializing [ 6525.679723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.692165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6527.497206] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6532.579914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.632769] Lustre: 240446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6532.778758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6534.364321] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6538.745986] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.178264] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.228272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.353298] Lustre: lustre-OST0000: new disk, initializing [ 6544.355300] Lustre: Skipped 1 previous similar message [ 6544.357870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6544.360147] Lustre: Skipped 2 previous similar messages [ 6544.362791] Lustre: 242410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6546.063781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6546.070299] Lustre: Skipped 1 previous similar message [ 6546.073376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6546.120947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6547.183224] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6552.840566] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6555.369954] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6555.503835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6557.997768] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6558.131394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6561.762419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6561.771555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.778027] LustreError: Skipped 1 previous similar message [ 6561.779635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6561.787143] Lustre: Skipped 4 previous similar messages [ 6561.800536] Lustre: Skipped 1 previous similar message [ 6565.542374] Lustre: server umount lustre-OST0000 complete [ 6568.929570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6572.001806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6572.002487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.011080] Lustre: Skipped 1 previous similar message [ 6572.022076] Lustre: Skipped 1 previous similar message [ 6573.850881] Lustre: server umount lustre-MDT0000 complete [ 6575.946277] LustreError: 241237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852407 with bad export cookie 1997461585960712936 [ 6575.947827] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6575.952907] LustreError: 241237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6576.128866] Lustre: server umount lustre-MDT0001 complete [ 6580.800646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.027676] LustreError: 244623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.068623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.074417] Lustre: Skipped 2 previous similar messages [ 6583.146935] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6586.340923] LustreError: 244624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.752780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.147623] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6594.372967] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6597.559677] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6601.286547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6604.317982] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6607.925873] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6754.274818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6754.284078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.289715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6754.292369] Lustre: Skipped 1 previous similar message [ 6768.096154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6768.164526] Lustre: server umount lustre-OST0000 complete [ 6776.049975] Lustre: server umount lustre-MDT0000 complete [ 6776.802155] LustreError: 245350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.810090] LustreError: 245350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6777.567848] LustreError: 247447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852609 with bad export cookie 1997461585960713818 [ 6777.571197] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.573759] LustreError: 247447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6777.721482] Lustre: server umount lustre-MDT0001 complete [ 6871.817964] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 6873.244678] Key type lgssc unregistered [ 6873.423468] LNet: 248233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6873.427205] LNetError: 248233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6873.435482] LNet: Removed LNI 192.168.206.131@tcp [ 6873.817151] Key type .llcrypt unregistered [ 6873.818577] Key type ._llcrypt unregistered [ 6884.442642] Key type ._llcrypt registered [ 6884.444051] Key type .llcrypt registered [ 6884.492426] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 6891.843622] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6892.315467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6892.327412] alg: No test for adler32 (adler32-zlib) [ 6893.211569] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 6893.320466] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 6894.912425] Key type lgssc registered [ 6895.437056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6916.610482] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 6922.060372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6922.070030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.179393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6923.195738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6923.240108] Lustre: lustre-MDT0000: new disk, initializing [ 6923.290533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.303665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6925.483608] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6930.319726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.365802] Lustre: 252271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.392668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6930.394910] Lustre: Skipped 1 previous similar message [ 6930.431122] Lustre: lustre-MDT0001: new disk, initializing [ 6930.470286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.485914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.490468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6932.095608] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6934.498355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6936.751726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.852976] Lustre: lustre-OST0000: new disk, initializing [ 6936.855453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6936.859264] Lustre: 253905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6936.894252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.858466] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6943.239275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6943.242888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6943.285295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6943.355515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.425426] Lustre: lustre-OST0001: new disk, initializing [ 6943.427790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6943.432071] Lustre: 254772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6943.477536] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6945.794280] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 6949.391550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6949.396558] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6949.447578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6951.947678] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6955.206518] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6959.482948] Lustre: server umount lustre-MDT0000 complete [ 6959.588733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.594733] Lustre: Skipped 3 previous similar messages [ 6961.133034] LustreError: 252263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852792 with bad export cookie 5592042695055825388 [ 6961.138675] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.140331] LustreError: 252263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6964.705333] LustreError: 252278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.707815] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.709835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6964.714551] LustreError: 252278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6964.721446] Lustre: Skipped 1 previous similar message [ 6966.754219] LustreError: 253076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.754522] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6966.770017] Lustre: Skipped 2 previous similar messages [ 6967.391352] Lustre: server umount lustre-MDT0001 complete [ 6969.445684] Lustre: server umount lustre-OST0000 complete [ 6971.404898] Lustre: server umount lustre-OST0001 complete [ 6978.921401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6984.423823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6988.694293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6992.871265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7000.489122] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7004.213853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.221810] Lustre: lustre-MDT0000: reset Object Index mappings [ 7004.384222] LustreError: 259768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7004.394620] LustreError: 259768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7004.406869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.975367] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7009.336813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.346765] Lustre: lustre-MDT0001: reset Object Index mappings [ 7011.285110] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7012.829593] Lustre: 260910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7016.397768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.596701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.601429] Lustre: Skipped 1 previous similar message [ 7019.255542] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7019.684843] LustreError: 261686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.526520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.739308] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7029.741804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7029.744628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7035.901210] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7037.464859] Lustre: 262779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7039.973281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.976075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.978852] Lustre: Skipped 1 previous similar message [ 7039.987381] Lustre: Skipped 3 previous similar messages [ 7045.090158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.096946] Lustre: Skipped 3 previous similar messages [ 7045.219257] Lustre: server umount lustre-MDT0000 complete [ 7046.902314] LustreError: 262782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852878 with bad export cookie 5592042695055832164 [ 7046.904646] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7046.906954] LustreError: 262782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7047.060726] Lustre: server umount lustre-MDT0001 complete [ 7058.442609] Lustre: server umount lustre-OST0000 complete [ 7069.349294] Lustre: server umount lustre-OST0001 complete [ 7076.454706] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:35:06 (1788852906) [ 7083.470790] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 7087.589073] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7101.881337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.985023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7101.998173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7102.036587] Lustre: lustre-MDT0000: new disk, initializing [ 7102.074355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7102.080861] Lustre: Skipped 1 previous similar message [ 7102.089685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7103.744247] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7108.098651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.146291] Lustre: 267027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.259979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7109.808097] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7113.751189] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7115.643378] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7118.118789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.246478] Lustre: lustre-OST0000: new disk, initializing [ 7118.248890] Lustre: Skipped 1 previous similar message [ 7118.251071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7118.253598] Lustre: Skipped 2 previous similar messages [ 7118.256733] Lustre: 268998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.125686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7120.129024] Lustre: Skipped 1 previous similar message [ 7120.130961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7120.162586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7120.632233] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7125.029822] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7127.097141] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7127.198657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7129.221611] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7129.312531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7130.594147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7130.595610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.599405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7130.611720] Lustre: Skipped 3 previous similar messages [ 7136.307752] Lustre: server umount lustre-OST0000 complete [ 7139.297209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.306170] LustreError: Skipped 1 previous similar message [ 7139.314522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.325129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.328678] Lustre: Skipped 3 previous similar messages [ 7144.678393] Lustre: server umount lustre-MDT0000 complete [ 7145.953260] LustreError: 267040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.963269] LustreError: 267040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7146.367079] LustreError: 267020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852978 with bad export cookie 5592042695055834236 [ 7146.368630] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.373688] LustreError: 267020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7146.519643] Lustre: server umount lustre-MDT0001 complete [ 7156.720877] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7162.788553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.996673] LustreError: 271924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.035187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.041308] Lustre: Skipped 2 previous similar messages [ 7165.939965] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7170.650167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.955770] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7174.158366] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7176.906601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.247376] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7182.445466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.523499] Lustre: lustre-OST0001: new disk, initializing [ 7182.526177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7182.530335] Lustre: 274232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7184.519637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7184.523212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7184.568927] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7184.729579] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7189.599284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7191.005711] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7193.227305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7193.246341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7196.640921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7196.647420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.653265] Lustre: Skipped 1 previous similar message [ 7196.655971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7196.659440] Lustre: Skipped 2 previous similar messages [ 7201.384276] Lustre: server umount lustre-MDT0000 complete [ 7203.139139] LustreError: 271906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853034 with bad export cookie 5592042695055835118 [ 7203.140508] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.146148] LustreError: 271906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7203.338451] Lustre: server umount lustre-MDT0001 complete [ 7215.777695] Lustre: server umount lustre-OST0000 complete [ 7227.928845] Lustre: server umount lustre-OST0001 complete [ 7235.071554] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7239.884380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.084706] LustreError: 277281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.096486] LustreError: 277281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7240.130464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.135081] Lustre: Skipped 3 previous similar messages [ 7242.123086] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7245.440327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.039076] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7248.227856] Lustre: 278419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7251.055125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.527257] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7257.262332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7259.774356] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7264.751456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7264.753801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7268.074297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7269.362904] Lustre: 280299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7271.250476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7271.260420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7274.811584] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:38:25 (1788853105) [ 7286.753214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7286.759270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7286.768728] Lustre: Skipped 3 previous similar messages [ 7286.772379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.775555] Lustre: Skipped 4 previous similar messages [ 7292.645425] Lustre: server umount lustre-MDT0000 complete [ 7294.028194] LustreError: 277261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853125 with bad export cookie 5592042695055841425 [ 7294.031984] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.033790] LustreError: 277261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7295.457526] LustreError: 279348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.464021] LustreError: 279348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7300.224712] Lustre: server umount lustre-MDT0001 complete [ 7307.844231] Lustre: server umount lustre-OST0000 complete [ 7315.356824] Lustre: server umount lustre-OST0001 complete [ 7317.886515] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 7321.861232] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7345.314969] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7350.260582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.362298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7350.375673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7350.414958] Lustre: lustre-MDT0000: new disk, initializing [ 7350.453813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7352.257119] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7357.693915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.770268] Lustre: 284529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.956085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7360.723783] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7367.018330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.165446] Lustre: 286161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7368.889806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7370.766240] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7377.112557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.205415] Lustre: 287031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7377.252107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7377.257183] Lustre: Skipped 6 previous similar messages [ 7378.325824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7380.066677] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7386.292939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7388.688781] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7388.692258] Lustre: Skipped 1 previous similar message [ 7397.675955] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:40:27 (1788853227) [ 7404.001901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7404.011969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.016685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7404.024199] Lustre: Skipped 8 previous similar messages [ 7404.031827] Lustre: Skipped 11 previous similar messages [ 7405.185415] Lustre: server umount lustre-MDT0000 complete [ 7407.362290] LustreError: 284521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853239 with bad export cookie 5592042695055849307 [ 7407.366507] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7407.368592] LustreError: 284521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7407.539752] Lustre: server umount lustre-MDT0001 complete [ 7419.741519] Lustre: server umount lustre-OST0000 complete [ 7431.719013] Lustre: server umount lustre-OST0001 complete [ 7435.427434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.606599] LustreError: 289279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.614643] LustreError: 289279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7437.135956] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7440.542724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.409025] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7445.387310] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7447.544896] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.256546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7452.967985] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7455.687604] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7467.825860] loop1: detected capacity change from 0 to 204800000 [ 7467.966760] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7467.971411] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7467.977609] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7470.250957] loop1: detected capacity change from 0 to 204800000 [ 7472.737777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.778368] LustreError: 289268:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs11912: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7472.786990] LustreError: 289268:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs11912 [ 7472.789916] LustreError: 289268:0:(mgs_handler.c:517:mgs_target_reg()) fs11912-MDT0001: Error checking target index: rc = -13 [ 7472.795273] LustreError: fs11912-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7472.800514] LustreError: 292993:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 7472.804665] LustreError: 292993:0:(tgt_mount.c:2132:server_put_super()) no obd fs11912-MDT0001 [ 7472.808073] LustreError: 292993:0:(tgt_mount.c:116:server_deregister_mount()) fs11912-MDT0001 not registered [ 7472.831575] Lustre: server umount fs11912-MDT0001 complete [ 7472.833508] LustreError: 292993:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7476.995757] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.040543] Lustre: Setting parameter fs11912-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11912-MDT0001 [ 7477.070244] Lustre: srv-fs11912-MDT0001: No data found on store. Initialize space. [ 7477.072969] Lustre: Skipped 4 previous similar messages [ 7477.112649] Lustre: fs11912-MDT0001: new disk, initializing [ 7477.114454] Lustre: Skipped 3 previous similar messages [ 7477.164737] Lustre: srv-fs11912-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7477.171386] Lustre: 293459:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11912-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7477.177257] LustreError: 293459:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11912-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7479.167539] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7483.051620] Lustre: server umount fs11912-MDT0001 complete [ 7489.297682] loop1: detected capacity change from 0 to 204800000 [ 7491.906688] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.044791] Lustre: 295024:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11912-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7492.048492] LustreError: 295024:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11912-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7493.832085] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7496.162266] Lustre: server umount fs11912-MDT0001 complete [ 7501.794424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7501.806146] Lustre: Skipped 2 previous similar messages [ 7506.428469] Lustre: server umount lustre-MDT0000 complete [ 7510.716060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.787834] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7512.860959] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7516.146987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7516.501536] loop1: detected capacity change from 0 to 204800000 [ 7519.416054] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.563660] Lustre: 297468:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11912-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7519.568198] LustreError: 297468:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11912-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7521.430735] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7523.453699] Lustre: server umount fs11912-MDT0001 complete [ 7533.540263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.549496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.552749] Lustre: Skipped 4 previous similar messages [ 7538.508492] Lustre: server umount lustre-OST0000 complete [ 7546.427257] Lustre: server umount lustre-MDT0000 complete [ 7548.126101] LustreError: 289261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853379 with bad export cookie 5592042695055851988 [ 7548.127389] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7548.135487] LustreError: 289261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7548.295991] Lustre: server umount lustre-MDT0001 complete [ 7553.249425] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 7554.740753] Key type lgssc unregistered [ 7554.919721] LNet: 299805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7554.925738] LNetError: 299805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7554.933975] LNet: Removed LNI 192.168.206.131@tcp [ 7555.360679] Key type .llcrypt unregistered [ 7555.363375] Key type ._llcrypt unregistered [ 7567.210368] Key type ._llcrypt registered [ 7567.212926] Key type .llcrypt registered [ 7567.284314] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7567.988543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.056522] alg: No test for adler32 (adler32-zlib) [ 7568.994601] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7569.152024] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 7570.771123] Key type lgssc registered [ 7571.416802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7576.212981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7576.221637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.491210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.050946] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7585.567860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.800033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7588.304819] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7593.527394] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7597.059598] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7600.841653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7601.115790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.184321] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.209580] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7605.550896] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7608.294939] LustreError: 303077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.101103] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7610.343120] LustreError: 303077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.351427] LustreError: 303077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7612.522644] LustreError: 303077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.901953] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.917138] LustreError: 302876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7621.691640] loop1: detected capacity change from 0 to 204800000 [ 7621.791789] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7621.799087] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7621.807577] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7624.702658] loop1: detected capacity change from 0 to 204800000 [ 7627.310054] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.386843] Lustre: fs29207-OST0000: new disk, initializing [ 7627.390861] Lustre: srv-fs29207-OST0000: No data found on store. Initialize space. [ 7627.397187] Lustre: 304782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7627.437192] Lustre: fs29207-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.139193] LustreError: 303076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.152109] LustreError: 303076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7630.187828] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7634.278338] Lustre: server umount fs29207-OST0000 complete [ 7639.552924] loop1: detected capacity change from 0 to 204800000 [ 7642.013396] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.159460] Lustre: fs29207-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.259109] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7646.183123] LustreError: 302875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.193377] LustreError: 302875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7649.512515] loop2: detected capacity change from 0 to 204800000 [ 7649.658267] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7649.663702] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7649.670230] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7652.453754] loop2: detected capacity change from 0 to 204800000 [ 7655.160700] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.207364] LustreError: fs29207-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7655.220498] LustreError: fs29207-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7655.226257] LustreError: 307818:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 7655.231804] LustreError: 307818:0:(tgt_mount.c:2132:server_put_super()) no obd fs29207-OST0001 [ 7655.240239] LustreError: 307818:0:(tgt_mount.c:116:server_deregister_mount()) fs29207-OST0001 not registered [ 7655.273500] Lustre: server umount fs29207-OST0001 complete [ 7655.276551] LustreError: 307818:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7659.802560] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.854305] Lustre: fs29207-OST0001: new disk, initializing [ 7659.856686] Lustre: srv-fs29207-OST0001: No data found on store. Initialize space. [ 7659.860247] Lustre: 308368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7659.889099] Lustre: fs29207-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7662.479037] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7666.904970] Lustre: server umount fs29207-OST0000 complete [ 7671.304547] Lustre: server umount fs29207-OST0001 complete [ 7677.091880] loop1: detected capacity change from 0 to 204800000 [ 7679.334944] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.342707] LustreError: 302874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 7679.514407] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.589363] Lustre: fs29207-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.445534] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7685.540566] loop2: detected capacity change from 0 to 204800000 [ 7687.144481] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7688.118359] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7690.605653] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7694.176265] Lustre: server umount fs29207-OST0000 complete [ 7697.381818] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7698.094756] Lustre: server umount fs29207-OST0001 complete [ 7705.571157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7705.574617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.583947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7707.617273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7707.621023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7707.622187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.413559] Lustre: server umount lustre-OST0000 complete [ 7712.737976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.740541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.744829] Lustre: Skipped 1 previous similar message [ 7712.747739] Lustre: Skipped 1 previous similar message [ 7717.342355] Lustre: server umount lustre-MDT0000 complete [ 7718.953969] LustreError: 301031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853550 with bad export cookie 14186721962308634684 [ 7718.956941] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.958992] LustreError: 301031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7719.101324] Lustre: server umount lustre-MDT0001 complete [ 7724.867336] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 7726.463428] Key type lgssc unregistered [ 7726.630581] LNet: 314280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7726.637331] LNetError: 314280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7726.648239] LNet: Removed LNI 192.168.206.131@tcp [ 7727.096204] Key type .llcrypt unregistered [ 7727.100454] Key type ._llcrypt unregistered [ 7743.776654] Key type ._llcrypt registered [ 7743.778590] Key type .llcrypt registered [ 7743.841500] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 7751.338303] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7751.696345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7751.728100] alg: No test for adler32 (adler32-zlib) [ 7752.609955] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7752.720313] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 7754.312369] Key type lgssc registered [ 7754.753817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7772.436328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7772.453172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.602552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7773.621506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7773.671746] Lustre: lustre-MDT0000: new disk, initializing [ 7773.714706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.727343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7776.022917] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7781.593091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.651390] Lustre: 318111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7781.689823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7781.693607] Lustre: Skipped 1 previous similar message [ 7781.754983] Lustre: lustre-MDT0001: new disk, initializing [ 7781.809496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.833938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7781.838703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7784.366489] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7789.798279] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7793.166420] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7796.794356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7796.983219] Lustre: lustre-OST0000: new disk, initializing [ 7796.985923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7796.989938] Lustre: 320077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7797.041506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.778329] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7805.460970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7805.466516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7805.531802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7805.846782] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.717777] Lustre: Mounted lustre-client - version 2.17.57_104_ga33798f [ 7807.154309] Lustre: Unmounted lustre-client [ 7812.067576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.072540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.074303] Lustre: Skipped 1 previous similar message [ 7817.187136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.191766] Lustre: Skipped 1 previous similar message [ 7822.306909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7822.309010] Lustre: Skipped 1 previous similar message [ 7825.376113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7825.459240] Lustre: server umount lustre-OST0000 complete [ 7827.941861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7827.946101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.951916] Lustre: Skipped 1 previous similar message [ 7827.955485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.545404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.545912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.623485] LustreError: 321485:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (0000000049db35d5) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7833.682586] Lustre: server umount lustre-MDT0000 complete [ 7835.999586] LustreError: 318101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853667 with bad export cookie 16486761688857079502 [ 7836.000836] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.006614] LustreError: 318101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7836.208422] Lustre: server umount lustre-MDT0001 complete [ 7841.715543] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 7842.986332] Key type lgssc unregistered [ 7857.184444] LNet: 1 local NIs in recovery (showing 1): 192.168.206.131@tcp [ 7857.441209] LNet: 322234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7857.447984] LNetError: 322234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7939.366947] LNet: Removed LNI 192.168.206.131@tcp [ 7939.847800] Key type .llcrypt unregistered [ 7939.849953] Key type ._llcrypt unregistered [ 7948.898816] Key type ._llcrypt registered [ 7948.901838] Key type .llcrypt registered [ 7948.973717] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 7957.928321] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 7958.312977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7958.325105] alg: No test for adler32 (adler32-zlib) [ 7959.215979] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7959.325615] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 7960.920583] Key type lgssc registered [ 7961.516807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7975.885225] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:50:06 (1788853806) [ 7978.656575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7978.665154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7979.780550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7979.796593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7979.834984] Lustre: lustre-MDT0000: new disk, initializing [ 7979.882692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.891801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7981.951703] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7987.437100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.486986] Lustre: 325429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7987.520551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7987.523138] Lustre: Skipped 1 previous similar message [ 7987.560791] Lustre: lustre-MDT0001: new disk, initializing [ 7987.600389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.618900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7987.623711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7989.265764] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 7993.922285] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7996.172724] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.750416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.866368] Lustre: lustre-OST0000: new disk, initializing [ 7998.868894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7998.872509] Lustre: 327394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7998.914547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.140544] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8005.599258] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8007.682149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8007.686523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8007.725125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8017.898564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.898732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.905866] LustreError: Skipped 1 previous similar message [ 8017.916478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8017.922641] Lustre: Skipped 1 previous similar message [ 8023.014891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8023.022114] Lustre: Skipped 1 previous similar message [ 8023.961681] Lustre: server umount lustre-OST0000 complete [ 8028.134516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.140558] Lustre: Skipped 1 previous similar message [ 8028.145491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.149436] Lustre: Skipped 1 previous similar message [ 8032.542647] Lustre: server umount lustre-MDT0000 complete [ 8034.915680] LustreError: 325419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853866 with bad export cookie 7825997168629558529 [ 8034.920993] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8034.921945] LustreError: 325419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8035.102892] Lustre: server umount lustre-MDT0001 complete [ 8040.822163] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8042.649487] Key type lgssc unregistered [ 8042.862422] LNet: 329300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8042.873298] LNetError: 329300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8042.902938] LNet: Removed LNI 192.168.206.131@tcp [ 8043.488903] Key type .llcrypt unregistered [ 8043.493041] Key type ._llcrypt unregistered [ 8054.134947] Key type ._llcrypt registered [ 8054.137188] Key type .llcrypt registered [ 8054.183397] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8054.714895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8054.723500] alg: No test for adler32 (adler32-zlib) [ 8055.613060] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8055.737750] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 8057.336416] Key type lgssc registered [ 8057.896848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8061.174265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8061.181638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.363701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.003287] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8067.669694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.882875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8069.785870] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8073.167497] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8075.711387] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.805417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.971498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.757511] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8085.056520] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8096.377806] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8097.016636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8098.785849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8098.789945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.798047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8102.773261] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 8102.882795] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.882717] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 8107.887960] Lustre: Skipped 3 previous similar messages [ 8111.009101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8111.013756] Lustre: Skipped 5 previous similar messages [ 8113.120169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8113.205360] Lustre: server umount lustre-MDT0000 complete [ 8115.003686] LustreError: 330529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853946 with bad export cookie 5654512238522046257 [ 8115.008569] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.011020] LustreError: 330529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8116.192854] LustreError: 333036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.193143] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.201443] LustreError: 333036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8116.216692] Lustre: Skipped 1 previous similar message [ 8116.219250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8116.222388] Lustre: Skipped 1 previous similar message [ 8118.118737] LustreError: 332987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.225308] LustreError: 333036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.228362] Lustre: server umount lustre-MDT0001 complete [ 8122.828811] Lustre: server umount lustre-OST0000 complete [ 8126.382447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.108689] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8129.293793] Lustre: 334725:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.131@tcp [ 8130.154320] Lustre: server umount MGS complete [ 8133.346932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.539640] LustreError: 335296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8133.562850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.044947] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8138.250633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.378077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8140.055729] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8143.385914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8145.759952] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8148.694634] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8149.370250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8151.771326] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8152.524503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8159.203495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8159.203596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.206292] LustreError: Skipped 1 previous similar message [ 8159.212762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8159.217473] Lustre: Skipped 3 previous similar messages [ 8160.534496] Lustre: server umount lustre-OST0000 complete [ 8164.321152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8164.321468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.335823] Lustre: Skipped 2 previous similar messages [ 8168.161402] Lustre: server umount lustre-MDT0000 complete [ 8169.441123] LustreError: 335296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.449187] LustreError: 335296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8169.661483] LustreError: 335277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854001 with bad export cookie 5654512238522047097 [ 8169.663811] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8169.669711] LustreError: 335277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8169.815327] Lustre: server umount lustre-MDT0001 complete [ 8174.299411] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8175.486703] Key type lgssc unregistered [ 8175.646265] LNet: 338728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8175.651514] LNetError: 338728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8175.661352] LNet: Removed LNI 192.168.206.131@tcp [ 8176.016657] Key type .llcrypt unregistered [ 8176.018218] Key type ._llcrypt unregistered [ 8183.520545] Key type ._llcrypt registered [ 8183.522483] Key type .llcrypt registered [ 8183.579824] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 8191.296359] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8191.735062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8191.784060] alg: No test for adler32 (adler32-zlib) [ 8192.714711] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8192.854144] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 8194.488419] Key type lgssc registered [ 8194.968694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8209.633658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8209.643671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.745548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8210.783472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8210.822128] Lustre: lustre-MDT0000: new disk, initializing [ 8210.851488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.862841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8212.600372] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8218.068905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.136193] Lustre: 341857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8218.180878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8218.183600] Lustre: Skipped 1 previous similar message [ 8218.228142] Lustre: lustre-MDT0001: new disk, initializing [ 8218.284137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8218.304361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8218.311212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8220.602854] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8225.910635] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8228.469882] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8232.064927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.214151] Lustre: lustre-OST0000: new disk, initializing [ 8232.218728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8232.228406] Lustre: 343825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8232.288257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.107852] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8240.149554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8240.157735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8240.171905] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.210904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8242.582078] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8242.670033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8244.541330] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8244.638315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8250.336942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8250.340317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.347200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8251.617120] Lustre: server umount lustre-OST0000 complete [ 8254.433085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.437313] LustreError: Skipped 1 previous similar message [ 8254.442290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.448871] Lustre: Skipped 1 previous similar message [ 8254.451284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.453583] Lustre: Skipped 1 previous similar message [ 8255.458561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.458720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.424596] Lustre: server umount lustre-MDT0000 complete [ 8260.908377] LustreError: 341894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854092 with bad export cookie 11387176908629648826 [ 8260.911854] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.914842] LustreError: 341894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8261.052503] Lustre: server umount lustre-MDT0001 complete [ 8264.956680] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:54:55 (1788854095) [ 8271.770624] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 8276.087362] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8290.280096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.374282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8290.389065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8290.429290] Lustre: lustre-MDT0000: new disk, initializing [ 8290.467766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.478909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8292.155171] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8297.266150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.318758] Lustre: 348574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8297.352174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8297.354960] Lustre: Skipped 1 previous similar message [ 8297.395279] Lustre: lustre-MDT0001: new disk, initializing [ 8297.438799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8297.455339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8297.461236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8299.439531] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8304.292529] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8306.848657] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8310.447636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.618191] Lustre: lustre-OST0000: new disk, initializing [ 8310.620827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8310.626295] Lustre: 350539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8310.682374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8313.751716] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8318.629636] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8320.534480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8320.541764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8320.593151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8340.963699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.965563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.972236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.972243] Lustre: Skipped 1 previous similar message [ 8340.973636] Lustre: Skipped 1 previous similar message [ 8345.383284] Lustre: server umount lustre-OST0000 complete [ 8348.641075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8348.647713] LustreError: Skipped 1 previous similar message [ 8348.652827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.661982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.666156] Lustre: Skipped 1 previous similar message [ 8353.395345] Lustre: server umount lustre-MDT0000 complete [ 8355.209149] LustreError: 348564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854186 with bad export cookie 11387176908629649694 [ 8355.213498] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8355.216816] LustreError: 348564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8355.366277] Lustre: server umount lustre-MDT0001 complete [ 8360.910083] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8362.512318] Key type lgssc unregistered [ 8362.665637] LNet: 352686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8362.669802] LNetError: 352686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8362.680517] LNet: Removed LNI 192.168.206.131@tcp [ 8363.082305] Key type .llcrypt unregistered [ 8363.084786] Key type ._llcrypt unregistered [ 8406.178632] Key type ._llcrypt registered [ 8406.180509] Key type .llcrypt registered [ 8406.235852] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8406.375302] Key type .llcrypt unregistered [ 8406.376961] Key type ._llcrypt unregistered [ 8440.757289] Key type ._llcrypt registered [ 8440.760216] Key type .llcrypt registered [ 8440.802444] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8440.955533] Key type .llcrypt unregistered [ 8440.957423] Key type ._llcrypt unregistered [ 8454.390883] Key type ._llcrypt registered [ 8454.392423] Key type .llcrypt registered [ 8454.451731] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8454.604513] Key type .llcrypt unregistered [ 8454.606439] Key type ._llcrypt unregistered [ 8470.350095] Key type ._llcrypt registered [ 8470.352360] Key type .llcrypt registered [ 8470.410258] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8470.555642] Key type .llcrypt unregistered [ 8470.559348] Key type ._llcrypt unregistered [ 8504.007646] Key type ._llcrypt registered [ 8504.009602] Key type .llcrypt registered [ 8504.054365] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8504.188621] Key type .llcrypt unregistered [ 8504.190421] Key type ._llcrypt unregistered [ 8517.817481] Key type ._llcrypt registered [ 8517.819656] Key type .llcrypt registered [ 8517.889876] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8518.033239] Key type .llcrypt unregistered [ 8518.036208] Key type ._llcrypt unregistered [ 8534.195705] Key type ._llcrypt registered [ 8534.197540] Key type .llcrypt registered [ 8534.253598] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8534.402238] Key type .llcrypt unregistered [ 8534.404102] Key type ._llcrypt unregistered [ 8548.957269] Key type ._llcrypt registered [ 8548.958786] Key type .llcrypt registered [ 8549.002431] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8549.122710] Key type .llcrypt unregistered [ 8549.124251] Key type ._llcrypt unregistered [ 8562.105226] Key type ._llcrypt registered [ 8562.106882] Key type .llcrypt registered [ 8562.161506] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8562.312471] Key type .llcrypt unregistered [ 8562.314050] Key type ._llcrypt unregistered [ 8578.119460] Key type ._llcrypt registered [ 8578.120902] Key type .llcrypt registered [ 8578.173776] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8578.337945] Key type .llcrypt unregistered [ 8578.340086] Key type ._llcrypt unregistered [ 8605.606249] Key type ._llcrypt registered [ 8605.607745] Key type .llcrypt registered [ 8605.655600] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8605.803328] Key type .llcrypt unregistered [ 8605.805500] Key type ._llcrypt unregistered [ 8620.677557] Key type ._llcrypt registered [ 8620.679043] Key type .llcrypt registered [ 8620.738853] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8620.873289] Key type .llcrypt unregistered [ 8620.875227] Key type ._llcrypt unregistered [ 8635.592426] Key type ._llcrypt registered [ 8635.594522] Key type .llcrypt registered [ 8635.661263] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8635.812224] Key type .llcrypt unregistered [ 8635.814118] Key type ._llcrypt unregistered [ 8660.180922] Key type ._llcrypt registered [ 8660.182376] Key type .llcrypt registered [ 8660.256271] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 8669.522123] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8670.046879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8670.056366] alg: No test for adler32 (adler32-zlib) [ 8671.011437] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8671.173282] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 8672.808131] Key type lgssc registered [ 8673.503269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8690.264058] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8690.278196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8691.421465] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8691.444386] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8691.495902] Lustre: tst250fs-MDT0000: new disk, initializing [ 8691.552643] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.563362] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8693.804881] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8699.767553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.832929] Lustre: 372568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8699.869457] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8699.872377] Lustre: Skipped 1 previous similar message [ 8699.913108] Lustre: tst250fs-MDT0001: new disk, initializing [ 8699.958945] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8699.978500] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8699.985828] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8701.886620] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8704.721440] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8707.879034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.995139] Lustre: tst250fs-OST0000: new disk, initializing [ 8707.997202] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8708.000835] Lustre: 374201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8708.055507] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.381078] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8708.385919] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8708.441721] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8710.769676] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8716.815528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.904975] Lustre: tst250fs-OST0001: new disk, initializing [ 8716.908527] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8716.912709] Lustre: 375072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8716.964635] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8717.796671] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8717.804628] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8717.864726] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8719.783290] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 8723.160764] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8735.714075] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8735.719314] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.728143] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.274029] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.278235] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.420842] Lustre: server umount tst250fs-MDT0000 complete [ 8743.052166] LustreError: 372562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854574 with bad export cookie 17672120961923399881 [ 8743.055857] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8743.057038] LustreError: 372562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8743.196505] Lustre: server umount tst250fs-MDT0001 complete [ 8755.036266] Lustre: server umount tst250fs-OST0000 complete [ 8766.905699] Lustre: server umount tst250fs-OST0001 complete [ 8775.618246] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 8779.534443] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8795.587788] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:03:45 (1788854625) [ 8796.414601] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8806.458315] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 8812.194281] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 8829.634816] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8578 sec ========== 04:04:19 (1788854659) [ 8830.513911] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:04:20 (1788854660) === [ 8832.138970] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:04:22 (1788854662) === [ 8851.217694] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 8852.630969] Key type lgssc unregistered [ 8852.791596] LNet: 384185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8852.797304] LNetError: 384185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8852.813221] LNet: Removed LNI 192.168.206.131@tcp [ 8853.208362] Key type .llcrypt unregistered [ 8853.213593] Key type ._llcrypt unregistered