[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 459079369 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007503] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014085] *** VALIDATE selinux *** [ 0.022189] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027211] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031091] *** VALIDATE tmpfs *** [ 0.033477] *** VALIDATE proc *** [ 0.034179] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036191] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040020] Spectre V2 : User space: Vulnerable [ 0.041004] Speculative Store Bypass: Vulnerable [ 0.043758] debug: unmapping init [mem 0xffffffffb3659000-0xffffffffb3660fff] [ 0.045160] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.049789] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.051916] ... event mask: 000000070000000f [ 0.052224] rcu: Hierarchical SRCU implementation. [ 0.054110] smp: Bringing up secondary CPUs ... [ 0.055388] x86: Booting SMP configuration: [ 0.056022] .... node #0, CPUs: #1 #2 #3 [ 0.060014] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144000] node 0 deferred pages initialised in 79ms [ 0.149030] devtmpfs: initialized [ 0.150175] x86/mm: Memory block size: 128MB [ 0.152565] gcov: version magic: 0x41383552 [ 0.153611] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161320] pinctrl core: initialized pinctrl subsystem [ 0.163164] [ 0.163743] ************************************************************* [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174010] ** ** [ 0.176011] ** This means that this kernel is built to expose internal ** [ 0.178011] ** IOMMU data structures, which may compromise security on ** [ 0.180013] ** your system. ** [ 0.183011] ** ** [ 0.185012] ** If you see this message and you are not debugging the ** [ 0.187013] ** kernel, report this immediately to your vendor! ** [ 0.189012] ** ** [ 0.191011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194012] ************************************************************* [ 0.196675] NET: Registered protocol family 16 [ 0.199417] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208454] cpuidle: using governor menu [ 0.209431] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212549] PCI: Using configuration type 1 for base access [ 0.215126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230049] cryptd: max_cpu_qlen set to 1000 [ 0.232247] ACPI: Added _OSI(Module Device) [ 0.234017] ACPI: Added _OSI(Processor Device) [ 0.236016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243703] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250422] ACPI: Interpreter enabled [ 0.252059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253013] ACPI: Using IOAPIC for interrupt routing [ 0.255139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257367] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268077] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283159] acpiphp: Slot [2] registered [ 0.285115] acpiphp: Slot [5] registered [ 0.287124] acpiphp: Slot [6] registered [ 0.288123] acpiphp: Slot [7] registered [ 0.291182] acpiphp: Slot [8] registered [ 0.293104] acpiphp: Slot [9] registered [ 0.294123] acpiphp: Slot [10] registered [ 0.296112] acpiphp: Slot [3] registered [ 0.297097] acpiphp: Slot [4] registered [ 0.299109] acpiphp: Slot [11] registered [ 0.300118] acpiphp: Slot [12] registered [ 0.302093] acpiphp: Slot [13] registered [ 0.303102] acpiphp: Slot [14] registered [ 0.305100] acpiphp: Slot [15] registered [ 0.306088] acpiphp: Slot [16] registered [ 0.307085] acpiphp: Slot [17] registered [ 0.309110] acpiphp: Slot [18] registered [ 0.310112] acpiphp: Slot [19] registered [ 0.312094] acpiphp: Slot [20] registered [ 0.313097] acpiphp: Slot [21] registered [ 0.315136] acpiphp: Slot [22] registered [ 0.316186] acpiphp: Slot [23] registered [ 0.318120] acpiphp: Slot [24] registered [ 0.320103] acpiphp: Slot [25] registered [ 0.321125] acpiphp: Slot [26] registered [ 0.323138] acpiphp: Slot [27] registered [ 0.324118] acpiphp: Slot [28] registered [ 0.326071] acpiphp: Slot [29] registered [ 0.327090] acpiphp: Slot [30] registered [ 0.328105] acpiphp: Slot [31] registered [ 0.330043] PCI host bridge to bus 0000:00 [ 0.331020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351582] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354394] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365699] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382640] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385060] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389868] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406946] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416743] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.426014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466161] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.500013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510609] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.568304] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.585014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.605071] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619902] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.628013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.653015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.663924] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.675014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704434] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709569] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.721138] iommu: Default domain type: Passthrough [ 0.722372] SCSI subsystem initialized [ 0.723165] ACPI: bus type USB registered [ 0.725104] usbcore: registered new interface driver usbfs [ 0.728185] usbcore: registered new interface driver hub [ 0.730079] usbcore: registered new device driver usb [ 0.732167] pps_core: LinuxPPS API ver. 1 registered [ 0.734016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.738080] PTP clock support registered [ 0.741069] EDAC MC: Ver: 3.0.0 [ 0.743111] PCI: Using ACPI for IRQ routing [ 0.745844] NetLabel: Initializing [ 0.747012] NetLabel: domain hash size = 128 [ 0.749012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751116] NetLabel: unlabeled traffic allowed by default [ 0.754146] vgaarb: loaded [ 0.756261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.758015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765558] clocksource: Switched to clocksource kvm-clock [ 0.871280] VFS: Disk quotas dquot_6.6.0 [ 0.873711] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876564] *** VALIDATE ramfs *** [ 0.878067] *** VALIDATE hugetlbfs *** [ 0.879757] pnp: PnP ACPI init [ 0.882168] pnp: PnP ACPI: found 6 devices [ 0.906397] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.911146] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.913341] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.915271] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.918415] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921097] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924033] NET: Registered protocol family 2 [ 0.927023] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.932208] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.936446] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.941319] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946600] TCP: Hash tables configured (established 65536 bind 65536) [ 0.950077] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.953609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956661] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959942] NET: Registered protocol family 1 [ 0.963182] RPC: Registered named UNIX socket transport module. [ 0.965664] RPC: Registered udp transport module. [ 0.967545] RPC: Registered tcp transport module. [ 0.969509] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.972134] NET: Registered protocol family 44 [ 0.974118] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976576] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.979069] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981601] PCI: CLS 0 bytes, default 64 [ 0.983475] Unpacking initramfs... [ 2.360241] debug: unmapping init [mem 0xffff9d043cc54000-0xffff9d043ffbffff] [ 2.364450] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.367294] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.370716] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.877783] Initialise system trusted keyrings [ 2.879150] Key type blacklist registered [ 2.881544] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.890675] zbud: loaded [ 2.894175] *** VALIDATE nfs *** [ 2.895687] *** VALIDATE nfs4 *** [ 2.899961] pstore: using deflate compression [ 2.903957] Platform Keyring initialized [ 3.009621] NET: Registered protocol family 38 [ 3.011826] Key type asymmetric registered [ 3.013422] Asymmetric key parser 'x509' registered [ 3.015248] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.018090] io scheduler mq-deadline registered [ 3.019540] io scheduler kyber registered [ 3.021891] io scheduler bfq registered [ 3.024279] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.028339] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.032122] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.035563] ACPI: Power Button [PWRF] [ 3.041710] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.049239] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.066206] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.072766] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.085323] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.112636] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.142420] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.148383] Non-volatile memory driver v1.3 [ 3.149688] Linux agpgart interface v0.103 [ 3.184458] virtio_blk virtio1: [vda] 146992 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.187874] vda: detected capacity change from 0 to 75259904 [ 3.207436] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.209911] vdb: detected capacity change from 0 to 1073741824 [ 3.228478] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.232138] vdc: detected capacity change from 0 to 2621440000 [ 3.249472] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252350] vdd: detected capacity change from 0 to 2621440000 [ 3.269206] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272555] vde: detected capacity change from 0 to 4294967296 [ 3.288435] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.292042] vdf: detected capacity change from 0 to 4294967296 [ 3.299668] libphy: Fixed MDIO Bus: probed [ 3.309136] usbcore: registered new interface driver usbserial_generic [ 3.312447] usbserial: USB Serial support registered for generic [ 3.315522] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.321966] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.324203] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.327124] mousedev: PS/2 mouse device common for all mice [ 3.330582] rtc_cmos 00:05: RTC can wake from S4 [ 3.334068] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.334384] rtc_cmos 00:05: registered as rtc0 [ 3.342195] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.342407] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.348381] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.350389] intel_pstate: CPU model not supported [ 3.359303] hid: raw HID events driver (C) Jiri Kosina [ 3.361949] usbcore: registered new interface driver usbhid [ 3.364599] usbhid: USB HID core driver [ 3.366569] drop_monitor: Initializing network drop monitor service [ 3.369273] Initializing XFRM netlink socket [ 3.371846] NET: Registered protocol family 10 [ 3.375516] Segment Routing with IPv6 [ 3.377400] NET: Registered protocol family 17 [ 3.379986] mpls_gso: MPLS GSO support [ 3.386091] RAS: Correctable Errors collector initialized. [ 3.388927] AVX version of gcm_enc/dec engaged. [ 3.391053] AES CTR mode by8 optimization enabled [ 3.483989] sched_clock: Marking stable (3483959682, 0)->(4390261706, -906302024) [ 3.488741] registered taskstats version 1 [ 3.491548] Loading compiled-in X.509 certificates [ 3.494132] zswap: loaded using pool lzo/zbud [ 3.526131] Key type big_key registered [ 3.538590] Key type encrypted registered [ 3.541089] ima: No TPM chip found, activating TPM-bypass! [ 3.543724] ima: Allocated hash algorithm: sha1 [ 3.546132] ima: No architecture policies found [ 3.548492] evm: Initialising EVM extended attributes: [ 3.550722] evm: security.selinux [ 3.551830] evm: security.ima [ 3.552739] evm: security.capability [ 3.554248] evm: HMAC attrs: 0x1 [ 3.557205] rtc_cmos 00:05: setting system clock to 2026-08-27 20:52:07 UTC (1787863927) [ 3.584836] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.588566] debug: unmapping init [mem 0xffffffffb3382000-0xffffffffb3658fff] [ 3.598125] Write protecting the kernel read-only data: 28672k [ 3.601965] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 3.605265] debug: unmapping init [mem 0xffffffffb2314000-0xffffffffb23fffff] [ 3.654307] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.663693] systemd[1]: Detected virtualization kvm. [ 3.665183] systemd[1]: Detected architecture x86-64. [ 3.667576] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.697754] systemd[1]: No hostname configured. [ 3.700111] systemd[1]: Set hostname to . [ 3.702807] random: systemd: uninitialized urandom read (16 bytes read) [ 3.706084] systemd[1]: Initializing machine ID from random generator. [ 3.759461] random: ln: uninitialized urandom read (6 bytes read) [ 3.851899] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855306] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.861345] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.866630] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.557910] device-mapper: uevent: version 1.0.3 [ 4.560642] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.376077] virtio_net virtio0 ens2: renamed from eth0 [ 5.397575] random: fast init done [ 5.462426] scsi host0: ata_piix [ 5.499421] scsi host1: ata_piix [ 5.501470] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.504436] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.376178] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.062729] random: crng init done [ 10.064620] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.653635] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.867219] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.121219] SELinux: Disabled at runtime. [ 12.188900] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.199425] systemd[1]: Detected virtualization kvm. [ 12.201967] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.761263] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.765778] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.771124] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.777518] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.782496] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.791214] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.798182] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.892142] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.237621] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.547424] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.563691] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.746253] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.760325] EDAC sbridge: Ver: 1.1.2 [ 15.235224] Key type dns_resolver registered [ 15.559214] NFS: Registering the id_resolver key type [ 15.561492] Key type id_resolver registered [ 15.563303] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg332-server login: [ 42.994905] libcfs: loading out-of-tree module taints kernel. [ 43.074985] Key type ._llcrypt registered [ 43.080607] Key type .llcrypt registered [ 43.266960] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 61.895178] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 63.642334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.665514] alg: No test for adler32 (adler32-zlib) [ 65.416281] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 66.606145] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 68.384859] Key type lgssc registered [ 70.617966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.357580] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.319423] hrtimer: interrupt took 7413386 ns [ 132.694589] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 146.227722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.247340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.534371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.569317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.646882] Lustre: lustre-MDT0000: new disk, initializing [ 147.751435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.784702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.350960] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.542601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.647950] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.711960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.717298] Lustre: Skipped 1 previous similar message [ 164.860687] Lustre: lustre-MDT0001: new disk, initializing [ 164.982039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.005055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.014907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.647416] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.498515] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.724517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.076399] Lustre: lustre-OST0000: new disk, initializing [ 184.082548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.095065] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.189600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.442117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.456561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.553613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.648350] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.857047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.992667] Lustre: lustre-OST0001: new disk, initializing [ 203.997853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.004972] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.084640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.758619] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.973074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.988356] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.028444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.787648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.392137] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.525732] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.080194] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing yml_node [ 242.089858] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 244.561549] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 246.966862] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 248.546118] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:56:11 EDT 2026 [ 264.787397] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 266.466945] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.592871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.600838] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.611552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.632431] Lustre: Skipped 1 previous similar message [ 282.665533] Lustre: Skipped 3 previous similar messages [ 287.717506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.732355] Lustre: Skipped 3 previous similar messages [ 288.478224] Lustre: server umount lustre-MDT0000 complete [ 291.857545] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864215 with bad export cookie 786851213685186294 [ 291.861963] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 291.887606] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 292.834752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.844340] Lustre: Skipped 2 previous similar messages [ 292.851798] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 292.859465] Lustre: Skipped 1 previous similar message [ 296.873388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.882746] Lustre: Skipped 1 previous similar message [ 298.420738] Lustre: server umount lustre-MDT0001 complete [ 302.571248] Lustre: server umount lustre-OST0000 complete [ 306.584910] Lustre: server umount lustre-OST0001 complete [ 313.359554] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 322.747860] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 358.311886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.529264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 358.556699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 358.623934] Lustre: lustre-MDT0000: new disk, initializing [ 358.691724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.711326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 362.541828] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 373.791834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.984376] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.101309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.113573] Lustre: Skipped 1 previous similar message [ 374.272937] Lustre: lustre-MDT0001: new disk, initializing [ 374.413798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.457249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.467100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.307643] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 390.452973] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 395.460445] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 402.251741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 402.553489] Lustre: lustre-OST0000: new disk, initializing [ 402.556733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 402.563278] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 402.682594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.411364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.419952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 404.500803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 409.068155] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 418.401887] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 423.423718] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 423.574696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.860227] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.250264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 435.177318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 435.190918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.214437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.226359] Lustre: Skipped 1 previous similar message [ 435.239560] Lustre: Skipped 1 previous similar message [ 438.812159] Lustre: server umount lustre-OST0000 complete [ 445.411697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.429414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.443387] Lustre: Skipped 1 previous similar message [ 448.831148] Lustre: server umount lustre-MDT0000 complete [ 450.535156] LustreError: 15080:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.550472] LustreError: 15080:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 452.333258] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864376 with bad export cookie 786851213685187827 [ 452.343595] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.685264] Lustre: server umount lustre-MDT0001 complete [ 457.939109] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:59:40 (1787864380) [ 475.861682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.279169] LustreError: 19917:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.337619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.907913] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 481.772426] LustreError: 19918:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.890153] LustreError: 19917:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.474864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.768687] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 501.117052] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.115292] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.251842] Lustre: Failing over lustre-MDT0000 [ 509.412054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.419530] LustreError: Skipped 1 previous similar message [ 509.430595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.448448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.464890] Lustre: Skipped 1 previous similar message [ 509.619292] Lustre: server umount lustre-MDT0000 complete [ 511.463692] LustreError: 20644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.487524] LustreError: 20644:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 523.752205] LustreError: 19914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.774047] LustreError: 19914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 527.843182] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864435/real 1787864435] req@ffff9d0383da1680 x1874711271335040/t0(0) o400->MGC192.168.203.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787864451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 527.883444] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.874717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.081601] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d0383c58780 x1874711271337856/t0(0) o250->MGC192.168.203.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 543.239591] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 543.714697] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 543.742855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 543.759060] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 552.426109] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 558.565179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.575519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 558.592229] Lustre: Skipped 2 previous similar messages [ 562.880701] Lustre: server umount lustre-MDT0000 complete [ 563.687565] LustreError: 19918:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.709047] LustreError: 19918:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 567.293507] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864491 with bad export cookie 786851213685189241 [ 567.304584] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 567.316237] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.594107] Lustre: server umount lustre-MDT0001 complete [ 577.985932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.449556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.458925] Lustre: Skipped 2 previous similar messages [ 583.012935] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 589.740580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.181325] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 601.846788] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.540669] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.748734] Lustre: Failing over lustre-MDT0000 [ 610.196646] Lustre: server umount lustre-MDT0000 complete [ 610.786028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 610.799907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.830516] LustreError: 23267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.867470] LustreError: 23267:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 629.155150] Lustre: 3638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864537/real 1787864537] req@ffff9d0383da1e00 x1874711271364224/t0(0) o400->MGC192.168.203.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787864553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 629.190903] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.152475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.457745] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d0383de4f00 x1874711271367424/t0(0) o250->MGC192.168.203.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.889806] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 644.221744] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 645.095418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 645.108843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 645.116768] Lustre: Skipped 1 previous similar message [ 645.144372] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 654.048617] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 659.428532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.452033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 659.458347] Lustre: Skipped 2 previous similar messages [ 659.479651] Lustre: Skipped 2 previous similar messages [ 672.224187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 672.698805] Lustre: server umount lustre-MDT0000 complete [ 675.810179] LustreError: 23266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.846809] LustreError: 23266:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 677.181076] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864601 with bad export cookie 786851213685190179 [ 677.183481] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.213751] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 677.663764] Lustre: server umount lustre-MDT0001 complete [ 687.195870] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:03:30 (1787864610) [ 703.080953] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 711.616408] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 743.697108] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 753.687891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.918724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 753.944990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 754.025715] Lustre: lustre-MDT0000: new disk, initializing [ 754.088217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.098218] Lustre: Skipped 2 previous similar messages [ 754.117622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 758.364521] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 769.076522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.223551] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 769.534732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 774.204297] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 783.823918] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 789.737238] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 797.123596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 797.405818] Lustre: 31610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 799.180431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 799.224445] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 799.312608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 799.332663] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 803.404972] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 813.425691] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 822.108568] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 840.168789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.169324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.193813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 840.199257] Lustre: Skipped 7 previous similar messages [ 841.821476] Lustre: server umount lustre-OST0000 complete [ 851.341369] Lustre: server umount lustre-MDT0000 complete [ 854.890682] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864778 with bad export cookie 786851213685190585 [ 854.895051] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.901186] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 855.258549] Lustre: server umount lustre-MDT0001 complete [ 865.532569] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 868.329515] Key type lgssc unregistered [ 868.623880] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.630733] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.654348] LNet: Removed LNI 192.168.203.132@tcp [ 869.790439] Key type .llcrypt unregistered [ 869.793068] Key type ._llcrypt unregistered [ 896.526695] Key type ._llcrypt registered [ 896.532829] Key type .llcrypt registered [ 896.627041] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 897.900345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.974260] alg: No test for adler32 (adler32-zlib) [ 899.041452] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 899.309343] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 900.984173] Key type lgssc registered [ 902.046359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.043590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.078672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.585957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.525896] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 921.065867] LustreError: 35009:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.088910] LustreError: 35009:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 924.040765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.387801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.643822] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 935.964906] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.873892] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 949.346745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.714300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.528851] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 964.601533] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.224878] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 982.412346] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:08:25 (1787864905) [ 984.330952] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 986.099769] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 992.663571] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:08:35 (1787864915) [ 1002.201833] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:08:44 (1787864924) [ 1003.325954] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1012.268415] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1012.276853] Lustre: Skipped 2 previous similar messages [ 1014.124136] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:08:57 (1787864937) [ 1018.015842] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1018.021079] Lustre: Skipped 3 previous similar messages [ 1037.233085] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1037.241347] Lustre: Skipped 6 previous similar messages [ 1040.108448] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:09:23 (1787864963) [ 1043.217386] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1047.196617] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1058.713877] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:09:40 (1787864980) [ 1061.111630] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1061.118788] Lustre: Skipped 3 previous similar messages [ 1074.722966] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:09:57 (1787864997) [ 1092.065605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.072281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.073783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.085453] Lustre: Skipped 1 previous similar message [ 1097.190514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.196781] Lustre: Skipped 2 previous similar messages [ 1102.314290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.321798] Lustre: Skipped 1 previous similar message [ 1104.352132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1104.554620] Lustre: server umount lustre-OST0000 complete [ 1108.964320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1108.974444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.019354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.546871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.890829] Lustre: server umount lustre-MDT0000 complete [ 1117.671027] LustreError: 35010:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.701522] LustreError: 35010:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1119.028892] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865042 with bad export cookie 1824691848352531504 [ 1119.036412] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.055524] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1119.409379] Lustre: server umount lustre-MDT0001 complete [ 1131.698807] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1134.818465] Key type lgssc unregistered [ 1135.112948] LNet: 42874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.118849] LNetError: 42874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.139166] LNet: Removed LNI 192.168.203.132@tcp [ 1136.334723] Key type .llcrypt unregistered [ 1136.337579] Key type ._llcrypt unregistered [ 1157.924287] Key type ._llcrypt registered [ 1157.941915] Key type .llcrypt registered [ 1158.115351] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 1176.064735] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1177.114060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.175599] alg: No test for adler32 (adler32-zlib) [ 1178.438950] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 1178.702759] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1180.368231] Key type lgssc registered [ 1181.292502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.087432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.156262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.485566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1221.572095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1221.738637] Lustre: lustre-MDT0000: new disk, initializing [ 1221.859012] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.886397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1226.873452] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1238.542881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.665317] Lustre: 46032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1238.785594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1238.791612] Lustre: Skipped 1 previous similar message [ 1238.891584] Lustre: lustre-MDT0001: new disk, initializing [ 1238.983373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.022599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1239.038352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1244.175675] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1255.103966] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1261.808231] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1269.425194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.777354] Lustre: lustre-OST0000: new disk, initializing [ 1269.784946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1269.802570] Lustre: 48000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1269.922201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.164597] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1280.086207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1280.108312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1280.222908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1287.215681] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.755086] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1292.982484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1298.633610] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1298.804562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.961909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.967980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.980020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.104164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.109987] Lustre: Skipped 2 previous similar messages [ 1311.206995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.216085] Lustre: Skipped 1 previous similar message [ 1315.297982] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.580318] Lustre: server umount lustre-OST0000 complete [ 1320.936724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1320.949625] LustreError: Skipped 1 previous similar message [ 1320.959653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.977105] Lustre: Skipped 1 previous similar message [ 1320.984791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.568916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.579544] Lustre: Skipped 3 previous similar messages [ 1334.240489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1334.366844] LustreError: 49542:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000009164e846) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 1336.479385] Lustre: server umount lustre-MDT0000 complete [ 1340.536727] LustreError: 46826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865264 with bad export cookie 8405550583464291318 [ 1340.538518] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.558813] LustreError: 46826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1340.979454] Lustre: server umount lustre-MDT0001 complete [ 1352.777517] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:14:35 (1787865275) [ 1360.598866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.149403] LustreError: 50465:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.184872] LustreError: 50465:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1361.281979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.264836] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1366.510145] LustreError: 50466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.619649] LustreError: 50465:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.014098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.270708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1381.586369] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1391.664421] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1399.255955] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1407.256891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.811266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.491971] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1422.720187] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1438.051784] LustreError: 53020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1438.949204] Lustre: Modifying parameter general.timeout=1 in log params [ 1440.632435] Lustre: Modifying parameter general.timeout=3 in log params [ 1440.637971] Lustre: Skipped 1 previous similar message [ 1442.643176] Lustre: Modifying parameter general.timeout=5 in log params [ 1442.651563] Lustre: Skipped 1 previous similar message [ 1446.782795] Lustre: Modifying parameter general.timeout=9 in log params [ 1446.786637] Lustre: Skipped 3 previous similar messages [ 1454.900695] Lustre: Modifying parameter general.timeout=18 in log params [ 1454.904681] Lustre: Skipped 8 previous similar messages [ 1471.010582] Lustre: Modifying parameter general.timeout=37 in log params [ 1471.017401] Lustre: Skipped 18 previous similar messages [ 1503.375193] Lustre: Modifying parameter general.timeout=72 in log params [ 1503.395283] Lustre: Skipped 34 previous similar messages [ 1569.313253] Lustre: Disabling parameter general.timeout= in log params [ 1569.316544] Lustre: Skipped 57 previous similar messages [ 1572.519783] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:18:15 (1787865495) [ 1575.116109] Lustre: 59975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1576.795677] Lustre: 60073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1580.182485] Lustre: 60270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1580.208094] Lustre: 60270:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1610.287757] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1610.293636] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1610.312759] LustreError: 61845:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1614.543331] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:18:57 (1787865537) [ 1630.700696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.714271] Lustre: Skipped 2 previous similar messages [ 1630.714306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.725569] Lustre: Skipped 4 previous similar messages [ 1634.859965] Lustre: server umount lustre-MDT0000 complete [ 1635.817472] LustreError: 50461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.843048] LustreError: 50461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1639.083137] LustreError: 53070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865563 with bad export cookie 8405550583464292193 [ 1639.096073] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.659877] Lustre: server umount lustre-MDT0001 complete [ 1654.672420] Lustre: server umount lustre-OST0000 complete [ 1684.742757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.074052] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1685.458575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.501399] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1699.356052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.529599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1699.581059] Lustre: 64386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1705.186271] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1713.380669] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1721.299726] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1729.243810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.510163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1729.803740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.808940] Lustre: Skipped 1 previous similar message [ 1736.533542] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1744.174193] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.798838] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1745.810783] Lustre: Skipped 23 previous similar messages [ 1745.865555] Lustre: 66957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1755.107867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.115066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.122727] Lustre: Skipped 1 previous similar message [ 1755.130154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.673138] Lustre: server umount lustre-OST0000 complete [ 1765.858338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.866463] Lustre: Skipped 1 previous similar message [ 1768.614856] Lustre: server umount lustre-MDT0000 complete [ 1770.980987] LustreError: 64394:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.006718] LustreError: 64394:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1772.193572] LustreError: 64379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865696 with bad export cookie 8405550583464295203 [ 1772.199032] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.206228] LustreError: 64379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1772.594658] Lustre: server umount lustre-MDT0001 complete [ 1785.095733] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1788.410502] Key type lgssc unregistered [ 1788.725962] LNet: 68169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.746140] LNetError: 68169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.766362] LNet: Removed LNI 192.168.203.132@tcp [ 1789.714400] Key type .llcrypt unregistered [ 1789.716824] Key type ._llcrypt unregistered [ 1817.977814] Key type ._llcrypt registered [ 1817.979514] Key type .llcrypt registered [ 1818.120873] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1819.710451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.805071] alg: No test for adler32 (adler32-zlib) [ 1820.904350] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 1821.144650] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1822.848186] Key type lgssc registered [ 1824.149532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.780569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1832.812811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.477875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.718941] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1844.719517] LustreError: 69585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.746384] LustreError: 69585:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1849.826432] LustreError: 69586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.852957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.447469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.823629] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1864.635386] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1871.225132] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1879.352428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.990174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.309575] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1894.179912] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1903.184104] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1915.881067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.891262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.907210] Lustre: Skipped 2 previous similar messages [ 1915.914270] Lustre: Skipped 2 previous similar messages [ 1918.394827] Lustre: server umount lustre-MDT0000 complete [ 1921.012437] LustreError: 70760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.026101] LustreError: 70760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1922.250779] LustreError: 69566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865846 with bad export cookie 470562932964479419 [ 1922.259559] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.266252] LustreError: 69566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1922.527427] Lustre: server umount lustre-MDT0001 complete [ 1937.022122] Lustre: server umount lustre-OST0000 complete [ 1965.428946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.742123] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1965.773480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1965.993582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.562776] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1978.741754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.822967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1978.861847] Lustre: 74456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.224781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.858086] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1991.478643] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1996.850399] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2004.190028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.445072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2004.896376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.900276] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2018.539334] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2020.435959] Lustre: 77032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2022.664344] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2028.513512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.527935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.549254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2029.029868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.163312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.168728] Lustre: Skipped 2 previous similar messages [ 2034.391675] Lustre: server umount lustre-OST0000 complete [ 2039.266826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.284089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.285731] Lustre: Skipped 2 previous similar messages [ 2039.303957] Lustre: Skipped 2 previous similar messages [ 2044.386339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.404067] Lustre: Skipped 1 previous similar message [ 2044.784091] Lustre: server umount lustre-MDT0000 complete [ 2048.643176] LustreError: 75203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865972 with bad export cookie 470562932964480259 [ 2048.645086] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.656339] LustreError: 75203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2049.037257] Lustre: server umount lustre-MDT0001 complete [ 2062.881742] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2065.769706] Key type lgssc unregistered [ 2066.164213] LNet: 78245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.178597] LNetError: 78245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.191496] LNet: Removed LNI 192.168.203.132@tcp [ 2067.487901] Key type .llcrypt unregistered [ 2067.490614] Key type ._llcrypt unregistered [ 2096.009195] Key type ._llcrypt registered [ 2096.012556] Key type .llcrypt registered [ 2096.121958] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2096.849610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.918054] alg: No test for adler32 (adler32-zlib) [ 2098.214784] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 2098.561263] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2100.320199] Key type lgssc registered [ 2101.508609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2109.097194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2109.135677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.574975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.789569] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2120.164163] LustreError: 79658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.183337] LustreError: 79658:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2122.525047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.114344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2127.848281] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2135.629798] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.952368] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2149.669275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2150.340948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.640654] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2165.203753] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2171.745205] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2172.869524] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2173.993787] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.092877] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.097483] Lustre: Skipped 1 previous similar message [ 2180.240427] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2180.247375] Lustre: Skipped 4 previous similar messages [ 2188.469644] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.484143] Lustre: Skipped 9 previous similar messages [ 2205.226970] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2205.232343] Lustre: Skipped 18 previous similar messages [ 2238.202831] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2238.212238] Lustre: Skipped 35 previous similar messages [ 2270.362280] Lustre: 87759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2273.015953] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:29:56 (1787866196) [ 2274.999393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2276.686118] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:29:59 (1787866199) [ 2283.489521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2283.503133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.519642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.539169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2285.547924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.551095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.583511] Lustre: Skipped 1 previous similar message [ 2287.396793] Lustre: server umount lustre-OST0000 complete [ 2292.193586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.205817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.234579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.780456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.787461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.538450] Lustre: server umount lustre-MDT0000 complete [ 2300.897083] LustreError: 79658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.938839] LustreError: 79658:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2301.330551] LustreError: 79640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866225 with bad export cookie 16522478977066085352 [ 2301.348957] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.356039] LustreError: 79640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2301.707365] Lustre: server umount lustre-MDT0001 complete [ 2312.754869] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2315.979488] Key type lgssc unregistered [ 2316.254970] LNet: 89012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.267604] LNetError: 89012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.293303] LNet: Removed LNI 192.168.203.132@tcp [ 2317.183539] Key type .llcrypt unregistered [ 2317.186272] Key type ._llcrypt unregistered [ 2322.129967] Key type ._llcrypt registered [ 2322.133693] Key type .llcrypt registered [ 2322.221232] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2327.202635] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules [ 2328.686387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.741322] alg: No test for adler32 (adler32-zlib) [ 2329.887439] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 2330.131541] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2331.824734] Key type lgssc registered [ 2333.136074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.121379] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2352.354839] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2356.111162] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2362.787463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.827541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.488457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.675429] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2374.626039] LustreError: 91239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.644606] LustreError: 91239:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2378.547436] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:31:41 (1787866301) [ 2379.750375] LustreError: 91240:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.406277] LustreError: 91271:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2383.414782] LustreError: 91271:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2383.427727] LustreError: 91271:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2383.751084] Lustre: server umount lustre-MDT0000 complete [ 2398.818682] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2401.543594] Key type lgssc unregistered [ 2401.845393] LNet: 92768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.855828] LNetError: 92768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.876198] LNet: Removed LNI 192.168.203.132@tcp [ 2402.725783] Key type .llcrypt unregistered [ 2402.730780] Key type ._llcrypt unregistered [ 2422.479687] Key type ._llcrypt registered [ 2422.485168] Key type .llcrypt registered [ 2422.668408] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2423.937306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2424.271419] alg: No test for adler32 (adler32-zlib) [ 2425.362570] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 2425.726310] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2427.384161] Key type lgssc registered [ 2428.679257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2436.910709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.940937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.448972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.476758] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2448.872415] LustreError: 93988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.889674] LustreError: 93988:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2452.101621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.656912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.652670] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2464.947982] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.860784] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2477.422942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.482529] LustreError: 95818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.934699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.330874] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2490.161847] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.427659] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:34:22 (1787866462) [ 2541.073234] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2542.868708] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:34:25 (1787866465) [ 2549.737465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.751192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.769173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.756558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.759401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.778642] Lustre: Skipped 1 previous similar message [ 2550.795977] Lustre: Skipped 2 previous similar messages [ 2553.580829] Lustre: server umount lustre-MDT0000 complete [ 2555.878677] LustreError: 95205:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.907400] LustreError: 95205:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2557.187771] LustreError: 93971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866481 with bad export cookie 4711700603490603900 [ 2557.200818] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.207573] LustreError: 93971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2557.526862] Lustre: server umount lustre-MDT0001 complete [ 2572.046855] Lustre: server umount lustre-OST0000 complete [ 2585.465388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.017390] LustreError: 98261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.247616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.236643] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2591.714052] LustreError: 98262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.673692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.407697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.256989] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2613.508949] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2619.791154] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2633.255000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.480806] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2633.492808] LustreError: 98251:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2633.513509] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2633.524495] LustreError: 100344:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2633.535202] LustreError: 100344:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2633.542458] LustreError: 100344:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2633.601574] Lustre: server umount lustre-OST0000 complete [ 2633.603540] LustreError: 100344:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2642.650813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.888545] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2642.900085] LustreError: 98250:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2642.913542] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2642.930425] LustreError: 100806:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2642.945450] LustreError: 100806:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2642.959139] LustreError: 100806:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2643.076780] Lustre: server umount lustre-OST0000 complete [ 2643.081561] LustreError: 100806:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2653.688660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.965668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2653.971974] Lustre: Found index 0 for lustre-OST0000, updating log [ 2653.979300] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2654.116412] Lustre: lustre-OST0000: new disk, initializing [ 2654.124450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.133631] Lustre: 101341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2654.491517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.368749] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2665.445779] Lustre: Failing over lustre-OST0000 [ 2665.728492] Lustre: server umount lustre-OST0000 complete [ 2670.434387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.436145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.454456] Lustre: Skipped 1 previous similar message [ 2675.682505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.687326] Lustre: Skipped 2 previous similar messages [ 2679.858179] Lustre: server umount lustre-MDT0000 complete [ 2680.801908] LustreError: 98258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.822759] LustreError: 98258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2683.734472] LustreError: 98244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866607 with bad export cookie 4711700603490606644 [ 2683.751400] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.753963] LustreError: 98244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2684.239280] Lustre: server umount lustre-MDT0001 complete [ 2693.179892] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:36:56 (1787866616) [ 2710.931291] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2721.024526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.360086] LustreError: 103689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.452563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.416982] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2734.213607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.359565] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2743.827621] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2751.931166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.296677] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2759.408919] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2759.417621] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2759.438488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2759.523227] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2764.786339] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2764.806242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.889229] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2766.910838] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2766.919267] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2766.942237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2766.947903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2768.130658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.256203] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2768.341428] Lustre: lustre-OST0001: new disk, initializing [ 2768.345226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2768.350582] Lustre: 106008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2768.614294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.622327] Lustre: Skipped 2 previous similar messages [ 2774.135348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2774.143086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2774.262526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2775.060626] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2784.647349] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2792.029460] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2798.058535] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.063841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2798.078977] Lustre: Skipped 2 previous similar messages [ 2798.086744] Lustre: Skipped 1 previous similar message [ 2808.301585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2808.304592] Lustre: Skipped 7 previous similar messages [ 2811.361778] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2811.575423] Lustre: server umount lustre-MDT0001 complete [ 2813.416544] LustreError: 103689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.458972] LustreError: 103689:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2822.117462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.259952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2822.265921] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2822.322744] Lustre: 103678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2823.808191] LustreError: 107559:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2823.821189] LustreError: 107559:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2823.832995] LustreError: 103681:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2823.939359] LustreError: 107581:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.132@tcp: cfg command failed: rc = -17 [ 2823.955236] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2823.955236] [ 2827.937107] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2828.290905] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2838.199870] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:39:21 (1787866761) [ 2856.720032] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2874.553780] Lustre: 109624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2897.959485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2901.645391] Lustre: 110760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2991.585391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.601489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.612171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.615636] Lustre: Skipped 1 previous similar message [ 2996.700875] Lustre: server umount lustre-MDT0000 complete [ 2997.218856] LustreError: 103685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.238898] LustreError: 103685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 3000.215828] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866924 with bad export cookie 4711700603490607568 [ 3000.216946] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.235486] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3000.526841] Lustre: server umount lustre-MDT0001 complete [ 3014.848723] Lustre: server umount lustre-OST0000 complete [ 3018.467397] Lustre: 93300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866926/real 1787866926] req@ffff9d04b8710000 x1874713745129088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787866942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3018.502731] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.515673] Lustre: Skipped 3 previous similar messages [ 3019.148658] Lustre: server umount lustre-OST0001 complete [ 3025.492516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3035.181464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3043.573496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3053.753570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3075.631657] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3086.968415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.997774] Lustre: lustre-MDT0000: reset Object Index mappings [ 3087.629533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.636512] Lustre: Skipped 1 previous similar message [ 3092.108490] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3100.849394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.887928] Lustre: lustre-MDT0001: reset Object Index mappings [ 3106.620710] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3109.710816] Lustre: 115859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3118.174945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.331441] LustreError: 116213:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.352692] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3126.362378] LustreError: 116213:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3126.922898] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3132.479092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3136.744631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.656542] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3142.661472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3143.929419] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3152.230609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3155.653219] Lustre: 117737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3210.503527] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:45:33 (1787867133) [ 3239.913395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.916661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.924953] Lustre: Skipped 2 previous similar messages [ 3239.933389] Lustre: Skipped 6 previous similar messages [ 3242.869881] Lustre: server umount lustre-MDT0000 complete [ 3245.938795] LustreError: 114691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867169 with bad export cookie 4711700603490654650 [ 3245.941322] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.951778] LustreError: 114691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3260.384319] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3260.781375] Lustre: server umount lustre-MDT0001 complete [ 3270.920047] Lustre: server umount lustre-OST0000 complete [ 3280.736355] Lustre: server umount lustre-OST0001 complete [ 3288.134790] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 3297.937664] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3336.666331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.953190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3336.976546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3337.056667] Lustre: lustre-MDT0000: new disk, initializing [ 3337.132223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.140864] Lustre: Skipped 3 previous similar messages [ 3337.151422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3341.684639] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3348.620644] Lustre: server umount lustre-MDT0000 complete [ 3359.116506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.339294] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3359.348178] Lustre: 122013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3359.358108] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3363.599993] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3368.375889] Lustre: server umount lustre-MDT0000 complete [ 3377.082375] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:48:20 (1787867300) [ 3378.858379] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3380.684595] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:48:23 (1787867303) [ 3398.988616] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3409.120669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.629517] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3421.236430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.365552] Lustre: 124065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3421.426298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3421.431537] Lustre: Skipped 1 previous similar message [ 3421.516599] Lustre: lustre-MDT0001: new disk, initializing [ 3421.648132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3421.656764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3425.644545] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3430.259435] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3437.265082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.532808] Lustre: lustre-OST0000: new disk, initializing [ 3437.535910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3437.541594] Lustre: 125641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3438.844571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3438.867513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3439.079138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3444.750907] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3455.657729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.894700] Lustre: lustre-OST0001: new disk, initializing [ 3455.897826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3455.903680] Lustre: 126511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3457.505847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3457.525690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3457.683516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3463.926453] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3474.430950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3477.935131] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3887.462382] Lustre: 132931:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3887.481229] Lustre: 132931:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3887.495318] Lustre: 132931:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3887.504203] Lustre: 132931:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3887.512367] Lustre: 132931:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3887.516654] Lustre: 132931:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4198.511955] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:01:59 (1787868119) [ 4215.272612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.279024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.279768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4215.291947] Lustre: Skipped 4 previous similar messages [ 4215.319198] Lustre: Skipped 9 previous similar messages [ 4219.802912] Lustre: server umount lustre-MDT0000 complete [ 4220.392762] LustreError: 127318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.412698] LustreError: 127318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 4223.208373] LustreError: 124057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868147 with bad export cookie 4711700603490724762 [ 4223.209051] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.220649] LustreError: 124057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4225.506389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.515245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4225.535473] Lustre: Skipped 3 previous similar messages [ 4225.548929] Lustre: Skipped 1 previous similar message [ 4229.924349] Lustre: server umount lustre-MDT0001 complete [ 4240.291603] Lustre: server umount lustre-OST0000 complete [ 4250.903606] Lustre: server umount lustre-OST0001 complete [ 4258.257829] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 4270.711486] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 4312.837855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.132794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4313.161116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4313.267844] Lustre: lustre-MDT0000: new disk, initializing [ 4313.341273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.344543] Lustre: Skipped 5 previous similar messages [ 4313.353886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4318.159652] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4329.407067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.616751] Lustre: 136007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4329.719588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4329.726699] Lustre: Skipped 1 previous similar message [ 4329.857303] Lustre: lustre-MDT0001: new disk, initializing [ 4330.006285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4330.013134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4334.758708] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4345.941804] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4354.738952] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4362.278555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.497893] Lustre: lustre-OST0000: new disk, initializing [ 4362.501797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4362.508805] Lustre: 137974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4362.580634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.586548] Lustre: Skipped 1 previous similar message [ 4364.406480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4364.433386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4364.552155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4370.327242] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4380.004487] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.421816] Lustre: lustre-MDD0000: changelog on [ 4394.329252] Lustre: lustre-MDD0001: changelog on [ 6395.927587] Lustre: lustre-MDD0001: changelog off [ 6398.222438] Lustre: 137169:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6399.126317] Lustre: lustre-MDD0000: changelog off [ 6407.651632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.657861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.677015] LustreError: Skipped 1 previous similar message [ 6407.682557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.682570] Lustre: Skipped 2 previous similar messages [ 6407.737854] Lustre: Skipped 1 previous similar message [ 6412.780554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.791554] Lustre: Skipped 2 previous similar messages [ 6413.758938] Lustre: server umount lustre-OST0000 complete [ 6418.921320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6418.948046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.972439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6423.011847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.129977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6428.147364] Lustre: Skipped 3 previous similar messages [ 6433.248125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6433.485180] Lustre: server umount lustre-MDT0000 complete [ 6437.357449] LustreError: 137169:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.378769] LustreError: 137169:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6437.975502] LustreError: 136000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870361 with bad export cookie 4711700603491549558 [ 6437.976431] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.989595] LustreError: 136000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6438.600255] Lustre: server umount lustre-MDT0001 complete [ 6451.871954] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6454.719583] Key type lgssc unregistered [ 6455.102754] LNet: 147130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6455.109958] LNetError: 147130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6455.134708] LNet: Removed LNI 192.168.203.132@tcp [ 6456.145203] Key type .llcrypt unregistered [ 6456.151483] Key type ._llcrypt unregistered [ 6478.163065] Key type ._llcrypt registered [ 6478.176951] Key type .llcrypt registered [ 6478.336718] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 6496.089733] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6497.357371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6497.477122] alg: No test for adler32 (adler32-zlib) [ 6498.692652] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 6499.012827] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6500.747039] Key type lgssc registered [ 6502.133896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6542.971672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6543.000168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.394476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6544.461257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6544.574776] Lustre: lustre-MDT0000: new disk, initializing [ 6544.666901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.706927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6549.584597] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6562.599043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.743984] Lustre: 150350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6562.834997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6562.839555] Lustre: Skipped 1 previous similar message [ 6562.946434] Lustre: lustre-MDT0001: new disk, initializing [ 6563.074397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6563.181113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6563.201600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6568.755904] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6577.997230] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6583.722635] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6590.345767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.635181] Lustre: lustre-OST0000: new disk, initializing [ 6590.643293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6590.656379] Lustre: 152317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6590.801485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.176584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6596.196825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6596.373534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6598.255892] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6610.168376] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6616.938278] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_module obdecho/obdecho [ 6619.339988] LustreError: 153324:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6619.352126] LustreError: 153324:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6621.235828] LustreError: 153420:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6637.539496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.544792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6637.567082] Lustre: Skipped 1 previous similar message [ 6637.582204] Lustre: Skipped 2 previous similar messages [ 6642.296284] Lustre: server umount lustre-MDT0000 complete [ 6646.115767] LustreError: 150340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870570 with bad export cookie 5182068991550620149 [ 6646.127996] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.133921] LustreError: 150340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.466827] Lustre: server umount lustre-MDT0001 complete [ 6661.180385] Lustre: server umount lustre-OST0000 complete [ 6683.998990] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 6697.067405] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6732.381415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.604347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6732.651268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6732.768360] Lustre: lustre-MDT0000: new disk, initializing [ 6732.873637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.890230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6737.236426] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6747.480829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.603928] Lustre: 157140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6747.647708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6747.651752] Lustre: Skipped 1 previous similar message [ 6747.726607] Lustre: lustre-MDT0001: new disk, initializing [ 6747.840660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6747.884057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6747.900139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6752.415276] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6761.772760] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.059477] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6775.332737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.642518] Lustre: lustre-OST0000: new disk, initializing [ 6775.646847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6775.666760] Lustre: 159106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6775.796962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.568154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6781.581449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6781.691741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6782.010531] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6791.397039] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.156478] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6796.336332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6801.057604] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6801.280757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6807.535780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.553829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.572028] Lustre: Skipped 1 previous similar message [ 6807.577959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.586867] Lustre: Skipped 1 previous similar message [ 6809.452377] Lustre: server umount lustre-OST0000 complete [ 6814.689573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.705201] LustreError: Skipped 1 previous similar message [ 6814.710613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.723570] Lustre: Skipped 1 previous similar message [ 6814.733698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.764916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.765655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.113923] Lustre: server umount lustre-MDT0000 complete [ 6822.626596] LustreError: 157130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870746 with bad export cookie 5182068991550621101 [ 6822.629790] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.641072] LustreError: 157130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6822.938553] Lustre: server umount lustre-MDT0001 complete [ 6826.979626] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:45:49 (1787870749) [ 6828.764624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6830.391508] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:45:53 (1787870753) [ 6838.748152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.227231] LustreError: 161519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.251213] LustreError: 161519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6839.317368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6844.306522] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6844.386856] LustreError: 161520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.507268] LustreError: 161519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.479124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.235730] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6869.220423] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6875.894198] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.937543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.266314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.271678] Lustre: Skipped 1 previous similar message [ 6889.691376] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6898.276470] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6905.313343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6905.321077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.332582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.335508] Lustre: Skipped 1 previous similar message [ 6910.853223] Lustre: server umount lustre-MDT0000 complete [ 6914.035604] LustreError: 161514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.059623] LustreError: 161514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6915.054415] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870838 with bad export cookie 5182068991550621983 [ 6915.066487] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.067025] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6919.144309] LustreError: 161514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.145061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.171965] LustreError: 161514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6919.192768] Lustre: Skipped 2 previous similar messages [ 6919.199616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6919.216339] Lustre: Skipped 4 previous similar messages [ 6929.376133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6929.623234] LustreError: 162953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.644097] LustreError: 162953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6929.815977] Lustre: server umount lustre-MDT0001 complete [ 6936.542726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6949.220329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6949.539589] LustreError: 165363:0:(llog.c:1655:llog_backup()) MGC192.168.203.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6949.546844] Lustre: 165363:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6956.008294] LustreError: 165368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.026296] LustreError: 165368:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6990.053393] LustreError: 165370:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.072761] LustreError: 165370:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 26 previous similar messages [ 7010.272311] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7010.282611] Lustre: 165363:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7051.257238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7055.573745] LustreError: 165385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.586425] LustreError: 165385:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 39 previous similar messages [ 7056.347216] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7056.362396] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7065.145952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.957977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7087.086477] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x47ea684ac2f1227f [ 7087.099654] Lustre: MGC192.168.203.132@tcp: Connection restored to 0@lo (at 0@lo) [ 7092.391361] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7092.720635] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7099.955097] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7101.625632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7107.506933] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7109.164151] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7128.545808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7128.554409] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.558168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.566210] Lustre: Skipped 1 previous similar message [ 7128.575864] Lustre: Skipped 6 previous similar messages [ 7134.332356] Lustre: server umount lustre-MDT0000 complete [ 7138.000246] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871061 with bad export cookie 5182068991550628479 [ 7138.001705] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.008891] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7138.345198] Lustre: server umount lustre-MDT0001 complete [ 7152.819780] Lustre: server umount lustre-OST0000 complete [ 7172.641741] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 7181.973923] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7217.004988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.254312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7217.278217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7217.377461] Lustre: lustre-MDT0000: new disk, initializing [ 7217.465796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.477967] Lustre: Skipped 1 previous similar message [ 7217.497492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7221.309388] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7231.617492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.711805] Lustre: 170471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7231.954328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7236.112868] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7245.106265] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7250.508849] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7256.740065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.973262] Lustre: lustre-OST0000: new disk, initializing [ 7256.980188] Lustre: Skipped 1 previous similar message [ 7256.991418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7257.000191] Lustre: Skipped 2 previous similar messages [ 7257.006891] Lustre: 172439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7264.471075] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7266.408872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7266.425424] Lustre: Skipped 1 previous similar message [ 7266.427410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7266.524058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7273.604584] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7278.114835] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7278.279467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7282.617820] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7282.852959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7287.265964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7287.266248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.277792] LustreError: Skipped 1 previous similar message [ 7287.278560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.292723] Lustre: Skipped 2 previous similar messages [ 7287.310939] Lustre: Skipped 4 previous similar messages [ 7299.040140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7299.269503] Lustre: server umount lustre-OST0000 complete [ 7303.655507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7309.969680] Lustre: server umount lustre-MDT0000 complete [ 7312.867405] LustreError: 170481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.894073] LustreError: 170481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 28 previous similar messages [ 7313.748516] LustreError: 170463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871237 with bad export cookie 5182068991550645279 [ 7313.753194] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.757355] LustreError: 170463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.149436] Lustre: server umount lustre-MDT0001 complete [ 7323.759795] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:54:06 (1787871246) [ 7330.724646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.744894] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7343.427745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.071447] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7355.047410] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7360.416582] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7367.254322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.343547] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7381.843952] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7394.236808] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:55:16 (1787871316) [ 7399.393537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7399.406668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.425952] Lustre: Skipped 2 previous similar messages [ 7399.431626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7399.439516] Lustre: Skipped 7 previous similar messages [ 7404.184613] Lustre: server umount lustre-OST0000 complete [ 7414.000977] Lustre: server umount lustre-MDT0000 complete [ 7417.484749] LustreError: 174833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871341 with bad export cookie 5182068991550646161 [ 7417.488260] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.505832] LustreError: 174833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7417.823568] Lustre: server umount lustre-MDT0001 complete [ 7428.566325] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7431.400750] Key type lgssc unregistered [ 7431.712965] LNet: 178587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.727218] LNetError: 178587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.757326] LNet: Removed LNI 192.168.203.132@tcp [ 7432.474368] Key type .llcrypt unregistered [ 7432.480063] Key type ._llcrypt unregistered [ 7452.266184] Key type ._llcrypt registered [ 7452.275255] Key type .llcrypt registered [ 7452.423983] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7453.987587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7454.080278] alg: No test for adler32 (adler32-zlib) [ 7455.169246] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 7455.478490] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 7457.176198] Key type lgssc registered [ 7458.412157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7466.257553] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7466.273385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.824782] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7486.944357] LustreError: 179829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 7492.065236] LustreError: 179829:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7492.091767] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7492.100267] LustreError: MGC192.168.203.132@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7492.117179] LustreError: 179829:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7492.125107] LustreError: 179829:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7492.130940] LustreError: 179829:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7492.218186] Lustre: server umount lustre-OST0000 complete [ 7492.228085] LustreError: 179829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7500.992063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.512089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.608817] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7513.058119] LustreError: 180330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.086688] LustreError: 180330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7516.090175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.467431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7521.019595] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7528.884417] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7534.480821] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7541.199874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.505282] LustreError: 182145:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7541.515576] Lustre: 182145:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.132@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7541.702854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7548.810893] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7556.935935] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7562.022582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7562.035568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7562.065584] Lustre: Skipped 2 previous similar messages [ 7562.085711] Lustre: Skipped 2 previous similar messages [ 7565.461750] Lustre: server umount lustre-MDT0000 complete [ 7567.342278] LustreError: 181057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.362709] LustreError: 181057:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7569.682500] LustreError: 180310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871493 with bad export cookie 4616464999416232456 [ 7569.705380] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.711648] LustreError: 180310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7570.296335] LustreError: 181059:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b8641ec0 x1874719019135232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7570.629957] Lustre: server umount lustre-MDT0001 complete [ 7585.478331] Lustre: server umount lustre-OST0000 complete [ 7595.213242] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 7605.835061] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7645.271389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.497134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.532959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7645.635732] Lustre: lustre-MDT0000: new disk, initializing [ 7645.687608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.707289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7650.753164] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7661.875425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.031117] Lustre: 185674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7662.094609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7662.100912] Lustre: Skipped 1 previous similar message [ 7662.192974] Lustre: lustre-MDT0001: new disk, initializing [ 7662.294432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7662.375608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7662.397027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7667.001684] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7676.650730] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7682.725693] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7689.644960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7689.874833] Lustre: lustre-OST0000: new disk, initializing [ 7689.879612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7689.885451] Lustre: 187643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7689.966437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.919358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7695.943435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7696.069601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7696.263399] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7706.952832] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7712.183576] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7712.427566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7717.786895] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7717.973931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7721.956801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.961759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.979741] LustreError: Skipped 1 previous similar message [ 7721.997514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7722.000590] Lustre: Skipped 1 previous similar message [ 7722.009159] Lustre: Skipped 1 previous similar message [ 7727.073812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.077058] Lustre: Skipped 1 previous similar message [ 7732.211455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.224821] Lustre: Skipped 1 previous similar message [ 7734.241906] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7734.448343] Lustre: server umount lustre-OST0000 complete [ 7739.361368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7739.367899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.408677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.436098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.707477] Lustre: server umount lustre-MDT0000 complete [ 7747.559530] LustreError: 188106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.576513] LustreError: 188106:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7748.340593] LustreError: 185666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871672 with bad export cookie 4616464999416233184 [ 7748.350885] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7748.355845] LustreError: 185666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7748.850384] Lustre: server umount lustre-MDT0001 complete [ 7758.189725] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:01:21 (1787871681) [ 7776.778484] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7779.430935] Key type lgssc unregistered [ 7779.692962] LNet: 190604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7779.701984] LNetError: 190604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7779.734361] LNet: Removed LNI 192.168.203.132@tcp [ 7780.534499] Key type .llcrypt unregistered [ 7780.536541] Key type ._llcrypt unregistered [ 7798.616248] Key type ._llcrypt registered [ 7798.621359] Key type .llcrypt registered [ 7798.757218] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7799.446953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7799.457900] alg: No test for adler32 (adler32-zlib) [ 7800.619335] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 7800.915299] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 7802.568508] Key type lgssc registered [ 7804.122435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.070620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7816.116283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.047222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.919132] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7827.428923] LustreError: 191823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.460201] LustreError: 191823:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7832.548222] LustreError: 191822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.881973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.217690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.309872] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7847.294358] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7853.193251] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7863.572972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.988619] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7863.994736] LustreError: 193688:0:(llog.c:1696:llog_backup()) MGC192.168.203.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7864.002600] Lustre: 193688:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.132@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7864.186706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.445635] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7880.309589] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7884.775057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.779287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.797970] Lustre: Skipped 2 previous similar messages [ 7884.812502] Lustre: Skipped 2 previous similar messages [ 7888.918550] Lustre: server umount lustre-MDT0000 complete [ 7889.892686] LustreError: 191819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.931094] LustreError: 191819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7892.417350] LustreError: 191802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871816 with bad export cookie 17400843644127032733 [ 7892.431416] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.434917] LustreError: 191802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7892.684786] Lustre: server umount lustre-MDT0001 complete [ 7907.613116] Lustre: server umount lustre-OST0000 complete [ 7916.796515] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 7928.356293] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7967.594266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.814682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7967.864633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7967.954651] Lustre: lustre-MDT0000: new disk, initializing [ 7968.030955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.050239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7972.339528] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7984.178415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.363364] Lustre: 197216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7984.433143] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7984.435460] Lustre: Skipped 1 previous similar message [ 7984.438192] LustreError: 197998:0:(llog.c:1696:llog_backup()) MGC192.168.203.132@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7984.443730] LustreError: 197998:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7984.448302] Lustre: 197998:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.132@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7984.469727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7984.478584] Lustre: Skipped 1 previous similar message [ 7984.606195] Lustre: lustre-MDT0001: new disk, initializing [ 7984.713592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7984.759921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7984.774884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7989.646344] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7999.535872] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8005.822932] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8013.150525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.444270] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8013.446264] Lustre: Skipped 1 previous similar message [ 8013.453336] LustreError: 199161:0:(llog.c:1696:llog_backup()) MGC192.168.203.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8013.464881] LustreError: 199161:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8013.476904] Lustre: 199161:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.132@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8013.487000] Lustre: 199161:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8013.510502] Lustre: lustre-OST0000: new disk, initializing [ 8013.517719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8013.529484] Lustre: 199185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8013.625982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.282775] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8022.093805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8022.105790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8022.204169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8029.370394] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.268797] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8034.482822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8039.163878] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8039.368701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8042.980400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.991719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.993060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.995454] LustreError: Skipped 1 previous similar message [ 8043.026748] Lustre: Skipped 1 previous similar message [ 8047.601848] Lustre: server umount lustre-OST0000 complete [ 8053.218993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8053.222437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8053.242446] Lustre: Skipped 1 previous similar message [ 8053.254980] Lustre: Skipped 2 previous similar messages [ 8057.671985] Lustre: server umount lustre-MDT0000 complete [ 8058.337048] LustreError: 197227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.361528] LustreError: 197227:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8061.278869] LustreError: 197209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871985 with bad export cookie 17400843644127033447 [ 8061.282463] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.291312] LustreError: 197209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8061.688397] Lustre: server umount lustre-MDT0001 complete [ 8070.501131] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:06:33 (1787871993) [ 8088.960103] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 8099.565086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.074459] LustreError: 202313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.113081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.157388] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8110.571863] LustreError: 202314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.591643] LustreError: 202314:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8114.792606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.044920] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8123.251800] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8129.577375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.150257] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8146.214073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8146.381804] Lustre: lustre-OST0001: new disk, initializing [ 8146.389489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8146.401912] Lustre: 204622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8146.507252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8146.513544] Lustre: Skipped 2 previous similar messages [ 8152.208335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8152.221357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8152.311300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8153.893474] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8167.134730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8175.830469] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8178.460300] Lustre: DEBUG MARKER: ADD OST3 [ 8186.331544] loop: module loaded [ 8186.480273] loop0: detected capacity change from 0 to 204800000 [ 8186.904449] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8186.949585] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8186.996376] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8194.669565] loop0: detected capacity change from 0 to 204800000 [ 8195.170884] Lustre: DEBUG MARKER: STOP OST3 [ 8201.876808] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.192199] Lustre: lustre-OST0003: new disk, initializing [ 8202.203164] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8202.228420] Lustre: 206659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8204.115332] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8204.248488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8204.258568] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8204.307606] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8205.649419] Lustre: Failing over lustre-OST0003 [ 8205.833571] Lustre: server umount lustre-OST0003 complete [ 8206.188743] LustreError: 206815:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8209.351974] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8209.397550] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8209.398862] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.414175] LustreError: 203798:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.437509] Lustre: Skipped 1 previous similar message [ 8209.457414] LustreError: 203798:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8219.659687] Lustre: DEBUG MARKER: START OST3 again [ 8227.662146] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.955057] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8227.967433] Lustre: Skipped 1 previous similar message [ 8227.984343] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8229.162540] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8229.306361] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8229.306768] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8234.944774] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8257.497385] Lustre: Failing over lustre-OST0003 [ 8257.603277] Lustre: server umount lustre-OST0003 complete [ 8258.546690] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.550486] LustreError: 203801:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.563514] Lustre: Skipped 1 previous similar message [ 8258.605326] LustreError: 203801:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 8279.018832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.022418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.023955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8279.053236] Lustre: Skipped 3 previous similar messages [ 8279.083502] Lustre: Skipped 3 previous similar messages [ 8282.800792] Lustre: server umount lustre-MDT0000 complete [ 8286.616060] LustreError: 203823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872210 with bad export cookie 17400843644127034329 [ 8286.621429] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8286.634608] LustreError: 203823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8289.252304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8293.040793] Lustre: server umount lustre-MDT0001 complete [ 8297.334393] Lustre: server umount lustre-OST0000 complete [ 8301.723420] Lustre: server umount lustre-OST0001 complete [ 8310.213180] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 8320.822892] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 8363.798598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.097820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8364.125053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8364.207896] Lustre: lustre-MDT0000: new disk, initializing [ 8364.267058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.295310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8370.340497] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8382.478190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.647253] Lustre: 212085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.767561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8382.774389] Lustre: Skipped 1 previous similar message [ 8382.908420] Lustre: lustre-MDT0001: new disk, initializing [ 8383.111862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8383.146394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8388.343618] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8398.896360] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8405.692019] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8413.318744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8413.726384] Lustre: 214054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8415.774989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8415.786595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8415.903650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8420.301933] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8430.894691] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8436.094485] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8436.358476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8441.272173] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8441.465673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8446.450757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.467130] Lustre: Skipped 2 previous similar messages [ 8446.478239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8446.484605] Lustre: Skipped 4 previous similar messages [ 8449.673834] Lustre: server umount lustre-OST0000 complete [ 8454.625759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8460.072413] Lustre: server umount lustre-MDT0000 complete [ 8461.793543] LustreError: 212094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8461.819837] LustreError: 212094:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 8464.212718] LustreError: 214053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872388 with bad export cookie 17400843644127036625 [ 8464.219913] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.227726] LustreError: 214053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8464.586389] Lustre: server umount lustre-MDT0001 complete [ 8468.873873] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:13:11 (1787872391) [ 8486.892731] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 8497.032746] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 8535.918249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8536.240154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8536.320648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8536.324331] Lustre: Skipped 1 previous similar message [ 8536.412920] Lustre: lustre-MDT0000: new disk, initializing [ 8536.418820] Lustre: Skipped 1 previous similar message [ 8536.628832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8541.555283] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8552.464291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.605609] Lustre: 218667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8553.022484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8558.193071] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8568.617472] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.055641] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8582.227502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.505934] Lustre: 220640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.948290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8589.090345] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8599.310722] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8605.190322] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8605.493348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8610.870747] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8611.149793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8614.884829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.889192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8614.890300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8614.890306] Lustre: Skipped 3 previous similar messages [ 8614.902889] Lustre: Skipped 4 previous similar messages [ 8619.701866] Lustre: server umount lustre-OST0000 complete [ 8624.614591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.226338] Lustre: server umount lustre-MDT0000 complete [ 8630.253099] LustreError: 218675:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.266084] LustreError: 218675:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8634.639816] LustreError: 218658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872558 with bad export cookie 17400843644127037500 [ 8634.654760] LustreError: 218658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8634.664511] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8635.275676] Lustre: server umount lustre-MDT0001 complete [ 8645.779409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.575643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.588298] Lustre: Skipped 5 previous similar messages [ 8652.990466] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8663.850798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.748232] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8678.854388] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8685.657612] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8693.173196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8700.739750] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8709.798694] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8726.500338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8726.500722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8726.503472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8726.503480] Lustre: Skipped 4 previous similar messages [ 8726.527822] Lustre: Skipped 3 previous similar messages [ 8732.508241] Lustre: server umount lustre-OST0000 complete [ 8743.075785] Lustre: server umount lustre-MDT0000 complete [ 8747.013968] LustreError: 222835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872670 with bad export cookie 17400843644127038389 [ 8747.024459] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8747.025331] LustreError: 222835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8747.443139] Lustre: server umount lustre-MDT0001 complete [ 8851.698209] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 8854.671571] Key type lgssc unregistered [ 8855.025659] LNet: 226547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8855.034673] LNetError: 226547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8855.052913] LNet: Removed LNI 192.168.203.132@tcp [ 8855.967929] Key type .llcrypt unregistered [ 8855.974679] Key type ._llcrypt unregistered [ 8882.234245] Key type ._llcrypt registered [ 8882.237744] Key type .llcrypt registered [ 8882.332703] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 8896.453444] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 8897.415305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8897.503886] alg: No test for adler32 (adler32-zlib) [ 8898.508438] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 8898.701602] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 8900.336194] Key type lgssc registered [ 8901.657720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8939.168627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8939.183901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.447156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8940.490801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8940.620252] Lustre: lustre-MDT0000: new disk, initializing [ 8940.724408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.748275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8945.673887] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8957.362187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8957.546742] Lustre: 229904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8957.665051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8957.673812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8957.694765] Lustre: Skipped 1 previous similar message [ 8957.813137] Lustre: lustre-MDT0001: new disk, initializing [ 8957.922639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8957.979828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8958.010102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8963.369483] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8973.108648] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8978.531961] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8986.567917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.829663] Lustre: lustre-OST0000: new disk, initializing [ 8986.837369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8986.844379] Lustre: 231869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.942153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.100411] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 8995.413133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8995.427243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8995.586475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9002.643298] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9006.936478] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9007.198762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9011.570979] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9011.742374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9016.292192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.298081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9016.301976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.312634] Lustre: Skipped 1 previous similar message [ 9016.332111] LustreError: Skipped 1 previous similar message [ 9019.611742] Lustre: server umount lustre-OST0000 complete [ 9024.481860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9024.489583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.505654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9024.509739] Lustre: Skipped 1 previous similar message [ 9026.535820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.536522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.569732] Lustre: Skipped 1 previous similar message [ 9031.651498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.656569] Lustre: Skipped 1 previous similar message [ 9037.280130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9037.404504] Lustre: server umount lustre-MDT0000 complete [ 9040.836072] LustreError: 229895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872964 with bad export cookie 11178250928338968088 [ 9040.837438] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.849640] LustreError: 229895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9041.242784] Lustre: server umount lustre-MDT0001 complete [ 9058.541503] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 9068.411731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.823601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.443574] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9074.150234] LustreError: 234804:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.163204] LustreError: 234804:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9078.245286] LustreError: 234803:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9079.921289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.179717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9084.303979] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9086.976579] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9092.627993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9093.044619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.723427] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9106.832988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9107.029412] Lustre: lustre-OST0001: new disk, initializing [ 9107.036196] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9107.043452] Lustre: 237112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9113.135160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9113.145882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9113.199440] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9113.255717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9126.448734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9135.577864] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9150.574398] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:24:32 (1787873072) [ 9154.536610] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.547440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9154.551227] Lustre: Skipped 3 previous similar messages [ 9154.573096] Lustre: Skipped 5 previous similar messages [ 9167.329341] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9167.657776] Lustre: server umount lustre-MDT0000 complete [ 9169.894976] LustreError: 235531:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.905241] LustreError: 235531:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9170.937742] LustreError: 234783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873094 with bad export cookie 11178250928338968963 [ 9170.955676] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.958164] LustreError: 234783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9171.288615] Lustre: server umount lustre-MDT0001 complete [ 9185.763939] Lustre: server umount lustre-OST0000 complete [ 9198.470680] Lustre: server umount lustre-OST0001 complete [ 9204.231829] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 9213.708501] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 9248.793544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.915698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9248.940573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.011159] Lustre: lustre-MDT0000: new disk, initializing [ 9249.075395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.080762] Lustre: Skipped 1 previous similar message [ 9249.089546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9252.692363] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9264.434518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.565067] Lustre: 241061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9264.636122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9264.646050] Lustre: Skipped 1 previous similar message [ 9264.735238] Lustre: lustre-MDT0001: new disk, initializing [ 9264.827907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9264.850939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9270.051725] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9280.335375] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9287.270163] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9294.748259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9295.012872] Lustre: lustre-OST0000: new disk, initializing [ 9295.022235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9295.038876] Lustre: 243027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9295.131768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.135779] Lustre: Skipped 1 previous similar message [ 9297.105323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9297.118922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9297.241859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9301.918216] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9311.875575] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9317.001338] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9317.248458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9321.768555] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9322.035867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9328.101532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.111806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.125497] Lustre: Skipped 1 previous similar message [ 9328.142211] Lustre: Skipped 9 previous similar messages [ 9338.343590] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9338.484953] Lustre: server umount lustre-OST0000 complete [ 9343.459040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.925824] Lustre: server umount lustre-MDT0000 complete [ 9352.594603] LustreError: 241051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873276 with bad export cookie 11178250928338971105 [ 9352.602253] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9352.607151] LustreError: 241051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9353.042446] Lustre: server umount lustre-MDT0001 complete [ 9361.408083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.868376] LustreError: 245244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9361.959505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.818793] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9367.022657] LustreError: 245245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9374.361288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.411973] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9387.484333] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9393.156476] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9399.841927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9406.936290] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9415.391357] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9563.109277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9563.112204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.124502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.134957] Lustre: Skipped 2 previous similar messages [ 9563.161307] Lustre: Skipped 9 previous similar messages [ 9567.838413] Lustre: server umount lustre-OST0000 complete [ 9586.144170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9586.366565] Lustre: server umount lustre-MDT0000 complete [ 9588.713850] LustreError: 245245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.752058] LustreError: 245245:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9590.393646] LustreError: 245226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873514 with bad export cookie 11178250928338971987 [ 9590.401315] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.406758] LustreError: 245226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9590.817972] Lustre: server umount lustre-MDT0001 complete [ 9685.836980] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 9688.731949] Key type lgssc unregistered [ 9689.092131] LNet: 248859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9689.096353] LNetError: 248859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9689.125188] LNet: Removed LNI 192.168.203.132@tcp [ 9690.203158] Key type .llcrypt unregistered [ 9690.210433] Key type ._llcrypt unregistered [ 9716.667292] Key type ._llcrypt registered [ 9716.671984] Key type .llcrypt registered [ 9716.754956] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 9731.947874] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 9732.843784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9733.045590] alg: No test for adler32 (adler32-zlib) [ 9734.062879] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 9734.294042] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 9736.080245] Key type lgssc registered [ 9737.203424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9785.630542] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 9796.638859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9796.694357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.944646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9797.968478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9798.028508] Lustre: lustre-MDT0000: new disk, initializing [ 9798.083710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.102214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9803.094250] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9814.403330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.542146] Lustre: 252932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9814.595878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9814.600041] Lustre: Skipped 1 previous similar message [ 9814.681632] Lustre: lustre-MDT0001: new disk, initializing [ 9814.782386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9814.824872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9814.847091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9819.311656] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9824.533150] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9831.980601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9832.331815] Lustre: lustre-OST0000: new disk, initializing [ 9832.335509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9832.346133] Lustre: 254564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9832.473738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9839.560117] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9840.729506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9840.740574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9840.849361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9850.169967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.372926] Lustre: lustre-OST0001: new disk, initializing [ 9850.377091] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9850.383213] Lustre: 255435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9850.499857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9856.806400] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 9860.180210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9860.205200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9860.334801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9868.644754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9875.643527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9891.299392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.312772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9891.329026] Lustre: Skipped 3 previous similar messages [ 9891.344375] Lustre: Skipped 3 previous similar messages [ 9893.422444] Lustre: server umount lustre-MDT0000 complete [ 9897.334258] LustreError: 253726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873821 with bad export cookie 14732947570305209884 [ 9897.339188] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.353401] LustreError: 253726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9897.630659] Lustre: server umount lustre-MDT0001 complete [ 9912.035190] Lustre: server umount lustre-OST0000 complete [ 9926.681758] Lustre: server umount lustre-OST0001 complete [ 9941.189682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9950.900175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9958.699909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9968.168829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9988.323759] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 9999.225883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9999.250791] Lustre: lustre-MDT0000: reset Object Index mappings [ 9999.617369] LustreError: 260449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.641705] LustreError: 260449:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9999.681300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10004.152344] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10004.965354] LustreError: 260450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10009.072804] LustreError: 260449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.745111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.777977] Lustre: lustre-MDT0001: reset Object Index mappings [10014.300375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10018.861445] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10021.680898] Lustre: 261591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10028.613942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.426772] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10039.222435] LustreError: 261947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.773128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10044.040947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10044.050398] Lustre: Skipped 1 previous similar message [10045.121106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10045.129967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10052.310146] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10061.242927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10065.288453] Lustre: 263463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10070.498803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10070.514827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.538766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.517239] Lustre: server umount lustre-MDT0000 complete [10076.132977] LustreError: 261461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.151719] LustreError: 261461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10076.279034] LustreError: 260429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874000 with bad export cookie 14732947570305216674 [10076.284030] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10076.291216] LustreError: 260429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10076.608437] Lustre: server umount lustre-MDT0001 complete [10090.710217] Lustre: server umount lustre-OST0000 complete [10105.069283] Lustre: server umount lustre-OST0001 complete [10125.044930] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:40:48 (1787874048) [10142.070663] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [10152.468033] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [10187.897453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.155475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10188.197769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10188.309181] Lustre: lustre-MDT0000: new disk, initializing [10188.429928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10188.461680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10193.081000] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10203.111197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10203.276606] Lustre: 267722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10203.351803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10203.359557] Lustre: Skipped 1 previous similar message [10203.435802] Lustre: lustre-MDT0001: new disk, initializing [10203.597521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10203.614734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10208.248766] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10217.141362] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10223.226756] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10229.740285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.963579] Lustre: lustre-OST0000: new disk, initializing [10229.967462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10229.973237] Lustre: 269686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10231.694182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10231.707915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10231.812874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10235.721526] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10244.998290] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10249.597462] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10249.778547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10253.835628] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10254.061859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10257.378164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10257.388317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.406043] Lustre: Skipped 3 previous similar messages [10257.413383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10257.417029] Lustre: Skipped 5 previous similar messages [10262.509635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10262.516621] Lustre: Skipped 1 previous similar message [10267.629477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10267.640194] Lustre: Skipped 1 previous similar message [10270.176425] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10270.390842] Lustre: server umount lustre-OST0000 complete [10275.298337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.307701] LustreError: Skipped 1 previous similar message [10275.315530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.338930] Lustre: Skipped 1 previous similar message [10277.859915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.862883] Lustre: Skipped 2 previous similar messages [10280.031621] Lustre: server umount lustre-MDT0000 complete [10282.977596] LustreError: 267733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.556520] LustreError: 267713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874207 with bad export cookie 14732947570305218767 [10283.566157] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.566298] LustreError: 267713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10283.899501] Lustre: server umount lustre-MDT0001 complete [10303.196587] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [10315.058976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.570389] LustreError: 272624:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.588601] LustreError: 272624:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10315.622198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10315.628035] Lustre: Skipped 2 previous similar messages [10320.063339] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10329.159381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10334.874348] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10338.266681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10345.863748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.763636] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10364.039790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.310591] Lustre: lustre-OST0001: new disk, initializing [10364.316688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10364.327874] Lustre: 274932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10366.382176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10366.411493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10366.586398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10371.213483] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10380.620092] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10384.658845] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10389.704947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10389.890329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10395.773321] Lustre: server umount lustre-MDT0000 complete [10396.132688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.139108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.162856] Lustre: Skipped 1 previous similar message [10396.170991] LustreError: 272625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.212447] LustreError: 272625:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10399.145719] LustreError: 275697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874323 with bad export cookie 14732947570305219649 [10399.150474] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.159353] LustreError: 275697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10399.552962] Lustre: server umount lustre-MDT0001 complete [10413.971051] Lustre: server umount lustre-OST0000 complete [10427.940722] Lustre: server umount lustre-OST0001 complete [10442.967262] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [10451.596561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.077969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10452.090877] Lustre: Skipped 3 previous similar messages [10456.412916] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10461.672194] LustreError: 277988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.705527] LustreError: 277988:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10465.117714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10470.576817] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10473.882740] Lustre: 279130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10480.052373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.568396] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10493.474888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10496.891046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10496.944932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10499.541970] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10508.088133] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10511.802578] Lustre: 281007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10516.759089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10516.871445] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10526.270785] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:47:29 (1787874449) [10542.561385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10542.570455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.585373] Lustre: Skipped 3 previous similar messages [10542.597085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10547.132689] Lustre: server umount lustre-MDT0000 complete [10550.486554] LustreError: 277968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874474 with bad export cookie 14732947570305226047 [10550.495291] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10550.503203] LustreError: 277968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10557.070206] Lustre: server umount lustre-MDT0001 complete [10576.352494] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10576.496370] Lustre: server umount lustre-OST0000 complete [10580.460241] Lustre: server umount lustre-OST0001 complete [10586.697791] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [10597.539157] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [10649.754770] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [10661.087980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.565937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10661.647522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10661.810605] Lustre: lustre-MDT0000: new disk, initializing [10661.953894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10667.334730] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10678.281123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.383784] Lustre: 285245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10678.576410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10682.611368] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10694.250682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.711239] Lustre: 286878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10696.843956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10701.205159] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10714.521216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.781729] Lustre: 287750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10714.878638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10714.881901] Lustre: Skipped 6 previous similar messages [10716.084099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10716.099279] Lustre: Skipped 1 previous similar message [10716.217638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10721.991341] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10731.502290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10735.224388] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10735.232154] Lustre: Skipped 1 previous similar message [10750.013988] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:51:12 (1787874672) [10754.496970] Lustre: server umount lustre-MDT0000 complete [10755.555164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10755.563241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.591766] Lustre: Skipped 5 previous similar messages [10755.608746] LustreError: 285257:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.636311] LustreError: 285257:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [10758.299216] LustreError: 285239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874682 with bad export cookie 14732947570305233866 [10758.313157] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.314803] LustreError: 285239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10758.636185] Lustre: server umount lustre-MDT0001 complete [10776.914541] Lustre: server umount lustre-OST0000 complete [10778.082164] Lustre: 250437:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874686/real 1787874686] req@ffff9d04b6190b40 x1874721409185024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787874702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10779.616176] Lustre: 250435:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874687/real 1787874687] req@ffff9d04879ff480 x1874721409185280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787874703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10781.026474] Lustre: server umount lustre-OST0001 complete [10791.248774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.640679] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10805.795227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10810.402417] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10817.843826] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10822.992313] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10829.303648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.637914] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10843.162295] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10858.859631] loop1: detected capacity change from 0 to 204800000 [10859.125283] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10859.139524] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10859.156063] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10864.667200] loop1: detected capacity change from 0 to 204800000 [10870.479596] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10870.563612] LustreError: 289988:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29931: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10870.576058] LustreError: 289988:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29931 [10870.583140] LustreError: 289988:0:(mgs_handler.c:517:mgs_target_reg()) fs29931-MDT0001: Error checking target index: rc = -13 [10870.597417] LustreError: fs29931-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10870.611215] LustreError: 293715:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10870.620806] LustreError: 293715:0:(tgt_mount.c:2129:server_put_super()) no obd fs29931-MDT0001 [10870.627089] LustreError: 293715:0:(tgt_mount.c:116:server_deregister_mount()) fs29931-MDT0001 not registered [10870.675487] Lustre: server umount fs29931-MDT0001 complete [10870.678229] LustreError: 293715:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10878.762089] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10878.845842] Lustre: Setting parameter fs29931-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29931-MDT0001 [10878.883656] Lustre: srv-fs29931-MDT0001: No data found on store. Initialize space. [10878.887303] Lustre: Skipped 4 previous similar messages [10878.939892] Lustre: fs29931-MDT0001: new disk, initializing [10878.941957] Lustre: Skipped 3 previous similar messages [10879.060485] Lustre: srv-fs29931-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10879.068704] Lustre: 294181:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29931-MDT0001-mdtlov: can't create new llog: rc = -115 [10879.079154] LustreError: 294181:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29931-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10882.870323] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10889.036890] Lustre: server umount fs29931-MDT0001 complete [10900.345621] loop1: detected capacity change from 0 to 204800000 [10905.200340] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.504431] Lustre: 295751:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29931-MDT0001-mdtlov: can't create new llog: rc = -115 [10905.515902] LustreError: 295751:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29931-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10909.476135] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10914.253343] Lustre: server umount fs29931-MDT0001 complete [10914.796537] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10924.003191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.022166] Lustre: Skipped 4 previous similar messages [10924.027329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10924.033968] Lustre: Skipped 8 previous similar messages [10929.686618] Lustre: server umount lustre-MDT0000 complete [10938.178823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.291273] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10943.198390] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10943.978138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10950.874922] loop1: detected capacity change from 0 to 204800000 [10956.361404] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.645115] Lustre: 298197:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29931-MDT0001-mdtlov: can't create new llog: rc = -115 [10956.653224] LustreError: 298197:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29931-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10961.098921] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [10965.775365] Lustre: server umount fs29931-MDT0001 complete [10988.716645] Lustre: server umount lustre-OST0000 complete [10995.182368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10995.186455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10995.193641] Lustre: Skipped 3 previous similar messages [10995.205763] Lustre: Skipped 4 previous similar messages [11002.467571] Lustre: server umount lustre-MDT0000 complete [11005.413674] Lustre: 250437:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874913/real 1787874913] req@ffff9d03829b30c0 x1874721409282304/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787874929 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [11006.912592] LustreError: 294728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874930 with bad export cookie 14732947570305236547 [11006.921726] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11006.928175] LustreError: 294728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11007.289949] Lustre: server umount lustre-MDT0001 complete [11020.057512] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [11023.486114] Key type lgssc unregistered [11023.841858] LNet: 300545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11023.854836] LNetError: 300545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11023.880326] LNet: Removed LNI 192.168.203.132@tcp [11024.964147] Key type .llcrypt unregistered [11024.966268] Key type ._llcrypt unregistered [11046.896081] Key type ._llcrypt registered [11046.904080] Key type .llcrypt registered [11047.039667] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [11048.616112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11048.667864] alg: No test for adler32 (adler32-zlib) [11049.731316] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [11049.943896] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [11051.640578] Key type lgssc registered [11053.076231] Lustre: Echo OBD driver; http://www.lustre.org/ [11061.666662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11061.707552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.383479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11068.524905] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11073.521313] LustreError: 301817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11073.544390] LustreError: 301817:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11076.962710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.500642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11081.752490] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11087.974068] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11092.271209] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11097.982028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.269765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11102.376661] LustreError: 303644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11103.375641] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11107.816496] LustreError: 303645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.310477] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11112.896212] LustreError: 303644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.914860] LustreError: 303644:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11118.307547] LustreError: 303645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11118.325355] LustreError: 303645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11126.695743] loop1: detected capacity change from 0 to 204800000 [11126.944473] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11126.959434] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11126.972560] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11128.547226] LustreError: 303645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.561695] LustreError: 303645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11132.160796] loop1: detected capacity change from 0 to 204800000 [11137.153021] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.312045] Lustre: fs30251-OST0000: new disk, initializing [11137.319987] Lustre: srv-fs30251-OST0000: No data found on store. Initialize space. [11137.331932] Lustre: 305550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11137.457317] Lustre: fs30251-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11144.620815] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11144.683299] LustreError: 303941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.707926] LustreError: 303941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [11154.170908] Lustre: server umount fs30251-OST0000 complete [11166.063892] loop1: detected capacity change from 0 to 204800000 [11171.166341] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11171.440234] Lustre: fs30251-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11177.181942] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11179.731458] LustreError: 303647:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.751311] LustreError: 303647:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [11185.650089] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11186.871301] loop2: detected capacity change from 0 to 204800000 [11187.100666] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11187.112669] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11187.139763] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11190.771159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11192.740714] loop2: detected capacity change from 0 to 204800000 [11198.120779] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11198.251102] LustreError: fs30251-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11198.261761] LustreError: fs30251-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11198.272196] LustreError: 308589:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11198.278550] LustreError: 308589:0:(tgt_mount.c:2129:server_put_super()) no obd fs30251-OST0001 [11198.285299] LustreError: 308589:0:(tgt_mount.c:116:server_deregister_mount()) fs30251-OST0001 not registered [11198.337060] Lustre: server umount fs30251-OST0001 complete [11198.340439] LustreError: 308589:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11209.342408] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.450966] Lustre: fs30251-OST0001: new disk, initializing [11209.456155] Lustre: srv-fs30251-OST0001: No data found on store. Initialize space. [11209.463722] Lustre: 309142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11209.560213] Lustre: fs30251-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11217.229820] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11225.072275] Lustre: server umount fs30251-OST0000 complete [11232.837364] Lustre: server umount fs30251-OST0001 complete [11243.427444] loop1: detected capacity change from 0 to 204800000 [11246.295049] LustreError: 309667:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.307515] LustreError: 309667:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [11248.654881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11248.930326] Lustre: fs30251-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.764856] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11260.966604] loop2: detected capacity change from 0 to 204800000 [11265.896415] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11271.825395] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11280.643525] Lustre: server umount fs30251-OST0000 complete [11288.744587] Lustre: server umount fs30251-OST0001 complete [11302.369508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11302.378364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.390869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11303.399679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11303.399935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.205802] Lustre: server umount lustre-OST0000 complete [11313.123084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11313.127447] LustreError: Skipped 1 previous similar message [11313.129457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11313.136681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11313.139201] Lustre: Skipped 1 previous similar message [11315.771960] Lustre: server umount lustre-MDT0000 complete [11319.400625] LustreError: 301797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875243 with bad export cookie 5837039693972954864 [11319.410933] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11319.416624] LustreError: 301797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11319.887768] Lustre: server umount lustre-MDT0001 complete [11331.449967] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [11334.054534] Key type lgssc unregistered [11334.264663] LNet: 315070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11334.275301] LNetError: 315070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11334.289762] LNet: Removed LNI 192.168.203.132@tcp [11335.226141] Key type .llcrypt unregistered [11335.227927] Key type ._llcrypt unregistered [11384.011667] Key type ._llcrypt registered [11384.022348] Key type .llcrypt registered [11384.213876] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [11408.873210] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [11410.156483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11410.406089] alg: No test for adler32 (adler32-zlib) [11411.640375] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [11412.019388] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [11413.712220] Key type lgssc registered [11415.107840] Lustre: Echo OBD driver; http://www.lustre.org/ [11449.857974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11449.918715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.142315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11451.188907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11451.382470] Lustre: lustre-MDT0000: new disk, initializing [11451.530383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11451.573850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11456.186934] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11466.638357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.755650] Lustre: 318944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11466.825843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11466.835481] Lustre: Skipped 1 previous similar message [11466.913209] Lustre: lustre-MDT0001: new disk, initializing [11467.022526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11467.073598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11467.089497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11471.349218] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11480.712154] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11486.566861] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11493.574346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11493.850408] Lustre: lustre-OST0000: new disk, initializing [11493.855628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11493.867671] Lustre: 320913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11493.995219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11500.112676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11500.118488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11500.251781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11500.344436] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11510.310721] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11512.531198] Lustre: Mounted lustre-client - version 2.17.57_106_g3758560 [11513.500184] Lustre: Unmounted lustre-client [11523.044307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.048406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11523.063210] Lustre: Skipped 1 previous similar message [11523.076915] Lustre: Skipped 1 previous similar message [11528.173657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.187973] Lustre: Skipped 1 previous similar message [11533.296558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11533.312673] Lustre: Skipped 1 previous similar message [11537.376802] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11537.551725] Lustre: server umount lustre-OST0000 complete [11543.524083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.542622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11548.642192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11548.648202] Lustre: Skipped 2 previous similar messages [11555.298507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11555.519870] Lustre: server umount lustre-MDT0000 complete [11559.149440] LustreError: 319738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875483 with bad export cookie 3563544444718367406 [11559.150373] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.162843] LustreError: 319738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11559.562606] Lustre: server umount lustre-MDT0001 complete [11568.736132] LNet: 1 local NIs in recovery (showing 1): 192.168.203.132@tcp [11569.645963] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [11572.969332] Key type lgssc unregistered [11573.232647] LNet: 323082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11573.250317] LNetError: 323082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11641.918855] LNet: Removed LNI 192.168.203.132@tcp [11643.063622] Key type .llcrypt unregistered [11643.065588] Key type ._llcrypt unregistered [11665.551688] Key type ._llcrypt registered [11665.555569] Key type .llcrypt registered [11665.770054] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [11683.761882] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [11684.528859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11684.624115] alg: No test for adler32 (adler32-zlib) [11685.759672] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [11685.971426] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [11687.640171] Key type lgssc registered [11688.885931] Lustre: Echo OBD driver; http://www.lustre.org/ [11719.039680] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:07:22 (1787875642) [11725.063189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11725.122211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.350924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11726.380241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11726.444524] Lustre: lustre-MDT0000: new disk, initializing [11726.548309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11726.570096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11731.243381] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11743.959201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.170250] Lustre: 326302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11744.230062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11744.238937] Lustre: Skipped 1 previous similar message [11744.403366] Lustre: lustre-MDT0001: new disk, initializing [11744.597484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11744.636930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11744.662121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11750.762097] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11762.190977] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11768.421900] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11775.419437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11775.674136] Lustre: lustre-OST0000: new disk, initializing [11775.678078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11775.684200] Lustre: 328270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11775.781244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11782.129159] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11783.239260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11783.249561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11783.341704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11791.397671] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.255460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11809.261900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.272497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.278803] Lustre: Skipped 1 previous similar message [11809.311110] Lustre: Skipped 1 previous similar message [11812.963324] Lustre: server umount lustre-OST0000 complete [11819.493282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.514502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.519741] Lustre: Skipped 1 previous similar message [11823.473823] Lustre: server umount lustre-MDT0000 complete [11827.087892] LustreError: 326294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875751 with bad export cookie 2963309913283841096 [11827.100038] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11827.102311] LustreError: 326294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11827.609469] Lustre: server umount lustre-MDT0001 complete [11838.996922] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [11841.357776] Key type lgssc unregistered [11841.669947] LNet: 330181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11841.678463] LNetError: 330181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11841.707447] LNet: Removed LNI 192.168.203.132@tcp [11842.381628] Key type .llcrypt unregistered [11842.383247] Key type ._llcrypt unregistered [11863.376601] Key type ._llcrypt registered [11863.379495] Key type .llcrypt registered [11863.519414] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [11864.237272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11864.251576] alg: No test for adler32 (adler32-zlib) [11865.489865] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [11865.962793] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [11867.780664] Key type lgssc registered [11869.191058] Lustre: Echo OBD driver; http://www.lustre.org/ [11879.220850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11879.239302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.667940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11885.428982] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11890.158973] LustreError: 331458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.185620] LustreError: 331458:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11893.607403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.910304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11897.904438] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11905.001864] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11910.903037] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11917.862163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11918.187352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11924.017873] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11931.043976] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11939.622069] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11940.903103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11944.930461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11944.945852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11944.963565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.965743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.967504] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.996770] Lustre: Skipped 1 previous similar message [11948.786253] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.32@tcp (stopping) [11948.790079] Lustre: Skipped 2 previous similar messages [11950.430838] Lustre: server umount lustre-MDT0000 complete [11951.089141] LustreError: 331459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.105390] LustreError: 331459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11953.877627] LustreError: 332517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.015779] LustreError: 331438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875877 with bad export cookie 14389409557048692458 [11954.023610] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11954.029103] LustreError: 331438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11956.193608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.195158] LustreError: 331454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11956.224644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11960.664825] Lustre: server umount lustre-MDT0001 complete [11964.734799] Lustre: server umount lustre-OST0000 complete [11975.188736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.300981] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [11983.792993] Lustre: 335638:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.132@tcp [11986.788927] Lustre: server umount MGS complete [11996.840948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11997.691247] LustreError: 336211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11997.725063] LustreError: 336211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11997.829968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12003.700585] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12007.394121] LustreError: 336212:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.413848] LustreError: 336212:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12013.704761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.166210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12018.118593] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12028.523695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12029.271890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12039.186836] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12050.678923] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12052.960553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12059.883950] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12062.353727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12069.345593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12069.367416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.404786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12069.413554] Lustre: Skipped 2 previous similar messages [12073.455179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12073.470063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.656512] Lustre: server umount lustre-OST0000 complete [12080.613052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12080.642435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12080.655663] Lustre: Skipped 2 previous similar messages [12083.688483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12083.730693] Lustre: Skipped 1 previous similar message [12086.658916] Lustre: server umount lustre-MDT0000 complete [12088.800761] LustreError: 336211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12088.830527] LustreError: 336211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12091.399104] LustreError: 336193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876015 with bad export cookie 14389409557048693298 [12091.407046] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12091.408253] LustreError: 336193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12091.902676] Lustre: server umount lustre-MDT0001 complete [12104.917757] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12108.615216] Key type lgssc unregistered [12108.964900] LNet: 339652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12108.973592] LNetError: 339652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12108.993655] LNet: Removed LNI 192.168.203.132@tcp [12109.997115] Key type .llcrypt unregistered [12110.000305] Key type ._llcrypt unregistered [12128.568159] Key type ._llcrypt registered [12128.569770] Key type .llcrypt registered [12128.640951] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [12142.908429] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [12144.100733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12144.130194] alg: No test for adler32 (adler32-zlib) [12145.189704] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [12145.455763] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [12147.128803] Key type lgssc registered [12148.111859] Lustre: Echo OBD driver; http://www.lustre.org/ [12186.157352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12186.170760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.376713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12187.395849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12187.466431] Lustre: lustre-MDT0000: new disk, initializing [12187.522144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12187.536172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12191.649867] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12202.061937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12202.199238] Lustre: 342815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12202.254808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12202.258478] Lustre: Skipped 1 previous similar message [12202.330742] Lustre: lustre-MDT0001: new disk, initializing [12202.419448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12202.462334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12202.476148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12207.372200] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12218.901502] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12226.384149] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12234.974805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12235.449655] Lustre: lustre-OST0000: new disk, initializing [12235.483911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12235.516591] Lustre: 344784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12235.732887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12242.655050] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12244.619355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12244.636856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12244.748455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12252.577539] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12257.455148] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12257.692711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12262.184117] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12262.418688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12265.442660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12265.460897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12265.468576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12265.473374] Lustre: Skipped 1 previous similar message [12270.646954] Lustre: server umount lustre-OST0000 complete [12275.682498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12275.713942] Lustre: Skipped 1 previous similar message [12275.722490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12280.804335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12280.814955] Lustre: Skipped 2 previous similar messages [12281.214425] Lustre: server umount lustre-MDT0000 complete [12285.889382] LustreError: 342808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876209 with bad export cookie 11183164022687328178 [12285.901411] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12285.902429] LustreError: 342808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12286.153196] Lustre: server umount lustre-MDT0001 complete [12297.221338] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:16:59 (1787876219) [12317.488745] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [12329.573575] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [12370.637797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12370.974355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12371.028132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12371.152551] Lustre: lustre-MDT0000: new disk, initializing [12371.209819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12371.228716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12375.538318] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12386.597752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.711523] Lustre: 349544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12386.761876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12386.775715] Lustre: Skipped 1 previous similar message [12386.884395] Lustre: lustre-MDT0001: new disk, initializing [12387.012607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12387.045585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12387.061314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12391.243582] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12401.859963] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12409.660378] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12417.220593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12417.654578] Lustre: lustre-OST0000: new disk, initializing [12417.658679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12417.669520] Lustre: 351511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12417.788586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12423.524670] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [12426.290360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12426.301138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12426.451541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12432.519792] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12477.937317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12477.946659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.954248] LustreError: Skipped 1 previous similar message [12477.973157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12477.982614] Lustre: Skipped 2 previous similar messages [12483.044806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12483.048197] Lustre: Skipped 1 previous similar message [12488.165487] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12488.451516] Lustre: server umount lustre-OST0000 complete [12493.286863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.296984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12493.303152] Lustre: Skipped 1 previous similar message [12493.317590] Lustre: Skipped 3 previous similar messages [12500.493256] Lustre: server umount lustre-MDT0000 complete [12503.526927] LustreError: 352232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.545755] LustreError: 352232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12504.779411] LustreError: 349535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876428 with bad export cookie 11183164022687329067 [12504.784076] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12504.793944] LustreError: 349535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12505.287663] Lustre: server umount lustre-MDT0001 complete [12519.674057] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12522.843495] Key type lgssc unregistered [12523.355075] LNet: 353695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12523.369608] LNetError: 353695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12523.419618] LNet: Removed LNI 192.168.203.132@tcp [12524.619633] Key type .llcrypt unregistered [12524.621512] Key type ._llcrypt unregistered [12631.062135] Key type ._llcrypt registered [12631.064217] Key type .llcrypt registered [12631.191891] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12631.610491] Key type .llcrypt unregistered [12631.616628] Key type ._llcrypt unregistered [12712.943433] Key type ._llcrypt registered [12712.949583] Key type .llcrypt registered [12713.106403] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12713.436300] Key type .llcrypt unregistered [12713.440449] Key type ._llcrypt unregistered [12749.536300] Key type ._llcrypt registered [12749.539224] Key type .llcrypt registered [12749.737832] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12750.298414] Key type .llcrypt unregistered [12750.303851] Key type ._llcrypt unregistered [12780.984407] Key type ._llcrypt registered [12780.986307] Key type .llcrypt registered [12781.147574] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12781.579318] Key type .llcrypt unregistered [12781.582928] Key type ._llcrypt unregistered [12871.117890] Key type ._llcrypt registered [12871.125028] Key type .llcrypt registered [12871.328227] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12871.770328] Key type .llcrypt unregistered [12871.774514] Key type ._llcrypt unregistered [12904.094951] Key type ._llcrypt registered [12904.096873] Key type .llcrypt registered [12904.213104] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12904.510367] Key type .llcrypt unregistered [12904.514132] Key type ._llcrypt unregistered [12936.543563] Key type ._llcrypt registered [12936.546578] Key type .llcrypt registered [12936.708824] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12937.050569] Key type .llcrypt unregistered [12937.058100] Key type ._llcrypt unregistered [12974.372196] Key type ._llcrypt registered [12974.375989] Key type .llcrypt registered [12974.546380] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [12975.093304] Key type .llcrypt unregistered [12975.100806] Key type ._llcrypt unregistered [13012.778992] Key type ._llcrypt registered [13012.784378] Key type .llcrypt registered [13013.035301] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13013.666510] Key type .llcrypt unregistered [13013.670878] Key type ._llcrypt unregistered [13047.306834] Key type ._llcrypt registered [13047.308719] Key type .llcrypt registered [13047.486465] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13047.996576] Key type .llcrypt unregistered [13047.999941] Key type ._llcrypt unregistered [13128.612978] Key type ._llcrypt registered [13128.614946] Key type .llcrypt registered [13128.755441] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13129.206114] Key type .llcrypt unregistered [13129.212418] Key type ._llcrypt unregistered [13164.359513] Key type ._llcrypt registered [13164.371414] Key type .llcrypt registered [13164.529695] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13165.007449] Key type .llcrypt unregistered [13165.015652] Key type ._llcrypt unregistered [13212.711495] Key type ._llcrypt registered [13212.714726] Key type .llcrypt registered [13212.880200] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13213.447170] Key type .llcrypt unregistered [13213.448685] Key type ._llcrypt unregistered [13277.309472] Key type ._llcrypt registered [13277.314375] Key type .llcrypt registered [13277.466500] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [13296.198751] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [13297.246957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13297.317988] alg: No test for adler32 (adler32-zlib) [13298.565774] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [13298.804721] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [13300.480186] Key type lgssc registered [13301.627761] Lustre: Echo OBD driver; http://www.lustre.org/ [13349.910675] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13349.982291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13351.255552] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13351.296628] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13351.438430] Lustre: tst250fs-MDT0000: new disk, initializing [13351.567790] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13351.603478] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13357.970446] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [13369.429439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.577555] Lustre: 374122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13369.649164] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13369.656030] Lustre: Skipped 1 previous similar message [13369.708937] Lustre: tst250fs-MDT0001: new disk, initializing [13369.809594] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13369.869982] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13369.884294] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13374.648824] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [13380.773832] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13389.123141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13389.481589] Lustre: tst250fs-OST0000: new disk, initializing [13389.491238] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13389.505554] Lustre: 375755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13389.644198] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13398.488242] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [13399.149828] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13399.180918] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13399.315423] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13410.484565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13410.687231] Lustre: tst250fs-OST0001: new disk, initializing [13410.690571] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13410.695678] Lustre: 376627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13410.808806] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13417.554255] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13417.566802] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13417.693624] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13418.111658] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [13424.416973] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13427.257881] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13427.263536] Lustre: Skipped 2 previous similar messages [13451.745559] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13451.755604] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13451.778609] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13453.803370] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13453.814176] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13453.830219] Lustre: Skipped 1 previous similar message [13453.854817] Lustre: Skipped 1 previous similar message [13457.834670] Lustre: server umount tst250fs-MDT0000 complete [13462.092851] LustreError: 374113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877386 with bad export cookie 6421932084923664445 [13462.102942] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13462.103661] LustreError: 374113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13462.548900] Lustre: server umount tst250fs-MDT0001 complete [13476.906983] Lustre: server umount tst250fs-OST0000 complete [13479.904336] Lustre: 372344:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787877387/real 1787877387] req@ffff9d0382b670c0 x1874725146008064/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787877403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13479.924469] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13479.935170] Lustre: Skipped 1 previous similar message [13481.621550] Lustre: server umount tst250fs-OST0001 complete [13508.375389] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [13516.265854] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [13547.328926] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:37:50 (1787877470) [13548.824674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13565.094356] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [13578.054369] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [13615.849517] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13365 sec ========= 20:38:58 (1787877538) [13618.316568] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:39:00 (1787877540) === [13623.139634] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:39:05 (1787877545) === [13666.035311] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [13668.889452] Key type lgssc unregistered [13669.217514] LNet: 385760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13669.227345] LNetError: 385760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13669.246391] LNet: Removed LNI 192.168.203.132@tcp [13670.154798] Key type .llcrypt unregistered [13670.156474] Key type ._llcrypt unregistered