[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458890564 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2846088K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006968] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011089] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.012820] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021162] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025580] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] *** VALIDATE tmpfs *** [ 0.029427] *** VALIDATE proc *** [ 0.030211] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035024] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039643] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.041163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042606] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043021] ... version: 2 [ 0.043937] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.044924] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046011] ... fixed-purpose events: 3 [ 0.046944] ... event mask: 000000070000000f [ 0.047297] rcu: Hierarchical SRCU implementation. [ 0.049508] smp: Bringing up secondary CPUs ... [ 0.050517] x86: Booting SMP configuration: [ 0.051021] .... node #0, CPUs: #1 #2 #3 [ 0.054133] smp: Brought up 1 node, 4 CPUs [ 0.055942] smpboot: Max logical packages: 1 [ 0.056013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137676] node 0 deferred pages initialised in 80ms [ 0.141011] devtmpfs: initialized [ 0.142223] x86/mm: Memory block size: 128MB [ 0.144579] gcov: version magic: 0x41383552 [ 0.146288] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148246] pinctrl core: initialized pinctrl subsystem [ 0.149155] [ 0.149539] ************************************************************* [ 0.150012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151009] ** ** [ 0.152010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153010] ** ** [ 0.154010] ** This means that this kernel is built to expose internal ** [ 0.155011] ** IOMMU data structures, which may compromise security on ** [ 0.156014] ** your system. ** [ 0.157010] ** ** [ 0.158013] ** If you see this message and you are not debugging the ** [ 0.159011] ** kernel, report this immediately to your vendor! ** [ 0.160010] ** ** [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162013] ************************************************************* [ 0.163699] NET: Registered protocol family 16 [ 0.164438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167397] cpuidle: using governor menu [ 0.168591] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171383] PCI: Using configuration type 1 for base access [ 0.173096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184112] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187102] cryptd: max_cpu_qlen set to 1000 [ 0.189208] ACPI: Added _OSI(Module Device) [ 0.191024] ACPI: Added _OSI(Processor Device) [ 0.192017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203293] ACPI: Interpreter enabled [ 0.204047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205013] ACPI: Using IOAPIC for interrupt routing [ 0.207093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218313] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.221044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.223018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229984] acpiphp: Slot [2] registered [ 0.232124] acpiphp: Slot [5] registered [ 0.233095] acpiphp: Slot [6] registered [ 0.234130] acpiphp: Slot [7] registered [ 0.236127] acpiphp: Slot [8] registered [ 0.237100] acpiphp: Slot [9] registered [ 0.238126] acpiphp: Slot [10] registered [ 0.240099] acpiphp: Slot [3] registered [ 0.241133] acpiphp: Slot [4] registered [ 0.243109] acpiphp: Slot [11] registered [ 0.244108] acpiphp: Slot [12] registered [ 0.245096] acpiphp: Slot [13] registered [ 0.246081] acpiphp: Slot [14] registered [ 0.248115] acpiphp: Slot [15] registered [ 0.249088] acpiphp: Slot [16] registered [ 0.250098] acpiphp: Slot [17] registered [ 0.252160] acpiphp: Slot [18] registered [ 0.254137] acpiphp: Slot [19] registered [ 0.255103] acpiphp: Slot [20] registered [ 0.256077] acpiphp: Slot [21] registered [ 0.258080] acpiphp: Slot [22] registered [ 0.259096] acpiphp: Slot [23] registered [ 0.261135] acpiphp: Slot [24] registered [ 0.262133] acpiphp: Slot [25] registered [ 0.264102] acpiphp: Slot [26] registered [ 0.265102] acpiphp: Slot [27] registered [ 0.267130] acpiphp: Slot [28] registered [ 0.268106] acpiphp: Slot [29] registered [ 0.270118] acpiphp: Slot [30] registered [ 0.271106] acpiphp: Slot [31] registered [ 0.273083] PCI host bridge to bus 0000:00 [ 0.274021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291986] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302696] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.308509] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.313020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.315018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.318352] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322804] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.331733] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.352014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363420] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.423040] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434528] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.483022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.496000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.506032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518036] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.543554] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.565019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603906] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.615029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.643033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653433] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.678018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.707000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.712480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716486] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.718291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.725036] iommu: Default domain type: Passthrough [ 0.726611] SCSI subsystem initialized [ 0.728127] ACPI: bus type USB registered [ 0.729126] usbcore: registered new interface driver usbfs [ 0.731203] usbcore: registered new interface driver hub [ 0.733127] usbcore: registered new device driver usb [ 0.735177] pps_core: LinuxPPS API ver. 1 registered [ 0.737011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740067] PTP clock support registered [ 0.743065] EDAC MC: Ver: 3.0.0 [ 0.744143] PCI: Using ACPI for IRQ routing [ 0.745000] NetLabel: Initializing [ 0.746010] NetLabel: domain hash size = 128 [ 0.747009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.749080] NetLabel: unlabeled traffic allowed by default [ 0.751156] vgaarb: loaded [ 0.753329] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.754016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757351] clocksource: Switched to clocksource kvm-clock [ 0.877072] VFS: Disk quotas dquot_6.6.0 [ 0.878883] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.882836] *** VALIDATE ramfs *** [ 0.885461] *** VALIDATE hugetlbfs *** [ 0.887865] pnp: PnP ACPI init [ 0.890629] pnp: PnP ACPI: found 6 devices [ 0.911460] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.915254] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.917555] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.919814] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.922169] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.924526] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.927373] NET: Registered protocol family 2 [ 0.930710] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.937410] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.942760] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.950519] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.956436] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960700] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964250] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967848] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971940] NET: Registered protocol family 1 [ 0.975275] RPC: Registered named UNIX socket transport module. [ 0.977862] RPC: Registered udp transport module. [ 0.979850] RPC: Registered tcp transport module. [ 0.982384] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985778] NET: Registered protocol family 44 [ 0.987453] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989092] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991143] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993568] PCI: CLS 0 bytes, default 64 [ 0.994788] Unpacking initramfs... [ 2.417840] debug: unmapping init [mem 0xffff97703cc54000-0xffff97703ffbffff] [ 2.422872] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.426128] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.429559] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.944899] Initialise system trusted keyrings [ 2.947365] Key type blacklist registered [ 2.949898] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.962075] zbud: loaded [ 2.966543] *** VALIDATE nfs *** [ 2.968393] *** VALIDATE nfs4 *** [ 2.970951] pstore: using deflate compression [ 2.975773] Platform Keyring initialized [ 3.083860] NET: Registered protocol family 38 [ 3.086647] Key type asymmetric registered [ 3.088170] Asymmetric key parser 'x509' registered [ 3.089844] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.092920] io scheduler mq-deadline registered [ 3.094410] io scheduler kyber registered [ 3.096115] io scheduler bfq registered [ 3.097485] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.100518] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.103331] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.105772] ACPI: Power Button [PWRF] [ 3.110574] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.117672] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.129641] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.136105] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.163286] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.193772] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.224981] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.228842] Non-volatile memory driver v1.3 [ 3.230720] Linux agpgart interface v0.103 [ 3.265637] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.268741] vda: detected capacity change from 0 to 69914624 [ 3.285185] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.289958] vdb: detected capacity change from 0 to 1073741824 [ 3.324047] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326865] vdc: detected capacity change from 0 to 2621440000 [ 3.343229] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346064] vdd: detected capacity change from 0 to 2621440000 [ 3.363053] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.365839] vde: detected capacity change from 0 to 4294967296 [ 3.385115] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.387479] vdf: detected capacity change from 0 to 4294967296 [ 3.393496] libphy: Fixed MDIO Bus: probed [ 3.399752] usbcore: registered new interface driver usbserial_generic [ 3.402558] usbserial: USB Serial support registered for generic [ 3.405106] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.408861] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.410725] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.413455] mousedev: PS/2 mouse device common for all mice [ 3.416105] rtc_cmos 00:05: RTC can wake from S4 [ 3.419405] rtc_cmos 00:05: registered as rtc0 [ 3.420645] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.421499] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.427923] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.428242] intel_pstate: CPU model not supported [ 3.433100] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.437858] hid: raw HID events driver (C) Jiri Kosina [ 3.440037] usbcore: registered new interface driver usbhid [ 3.442273] usbhid: USB HID core driver [ 3.443904] drop_monitor: Initializing network drop monitor service [ 3.446444] Initializing XFRM netlink socket [ 3.448596] NET: Registered protocol family 10 [ 3.451953] Segment Routing with IPv6 [ 3.453549] NET: Registered protocol family 17 [ 3.455655] mpls_gso: MPLS GSO support [ 3.462467] RAS: Correctable Errors collector initialized. [ 3.464453] AVX version of gcm_enc/dec engaged. [ 3.465947] AES CTR mode by8 optimization enabled [ 3.543373] sched_clock: Marking stable (3543332930, 0)->(4394724143, -851391213) [ 3.547431] registered taskstats version 1 [ 3.549302] Loading compiled-in X.509 certificates [ 3.551215] zswap: loaded using pool lzo/zbud [ 3.576467] Key type big_key registered [ 3.589576] Key type encrypted registered [ 3.591291] ima: No TPM chip found, activating TPM-bypass! [ 3.593458] ima: Allocated hash algorithm: sha1 [ 3.595183] ima: No architecture policies found [ 3.597066] evm: Initialising EVM extended attributes: [ 3.599009] evm: security.selinux [ 3.600322] evm: security.ima [ 3.601465] evm: security.capability [ 3.602865] evm: HMAC attrs: 0x1 [ 3.605226] rtc_cmos 00:05: setting system clock to 2026-06-11 23:26:29 UTC (1781220389) [ 3.611958] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 3.615019] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 3.622361] Write protecting the kernel read-only data: 28672k [ 3.625503] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.628493] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 3.658474] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.666485] systemd[1]: Detected virtualization kvm. [ 3.668271] systemd[1]: Detected architecture x86-64. [ 3.669716] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.697363] systemd[1]: No hostname configured. [ 3.699205] systemd[1]: Set hostname to . [ 3.701414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.704015] systemd[1]: Initializing machine ID from random generator. [ 3.760194] random: ln: uninitialized urandom read (6 bytes read) [ 3.847182] random: systemd: uninitialized urandom read (16 bytes read) [ 3.850574] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.855692] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.864901] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.445563] device-mapper: uevent: version 1.0.3 [ 4.447601] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.981873] virtio_net virtio0 ens2: renamed from eth0 [ 5.997974] random: fast init done [ 6.214840] scsi host0: ata_piix [ 6.356131] scsi host1: ata_piix [ 6.360407] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.366728] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.133508] random: crng init done [ 11.138178] random: 7 urandom warning(s) missed due to ratelimiting [ 13.783066] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.869781] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.678603] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.592766] SELinux: Disabled at runtime. [ 19.697067] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.725331] systemd[1]: Detected virtualization kvm. [ 19.739909] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.186549] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.189860] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.197589] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.204749] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.210468] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.220212] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.234943] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug F[ 21.690733] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ile System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.864674] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.825855] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.927740] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.387408] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.508915] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 30.935440] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 31.777654] NFS: Registering the id_resolver key type [ 31.794642] Key type id_resolver registered [ 31.796390] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg248-server login: [ 97.964513] libcfs: loading out-of-tree module taints kernel. [ 98.088757] Key type ._llcrypt registered [ 98.098253] Key type .llcrypt registered [ 98.230349] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 119.418790] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 121.577948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.592080] alg: No test for adler32 (adler32-zlib) [ 123.634317] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 124.679754] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 126.431266] Key type lgssc registered [ 128.259430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.754026] hrtimer: interrupt took 7015952 ns [ 147.293586] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.881524] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 206.575489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 206.640965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.904520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 207.928991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.040569] Lustre: lustre-MDT0000: new disk, initializing [ 208.147503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.163985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 212.756710] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.255384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.500574] Lustre: 6519:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.585640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.591286] Lustre: Skipped 1 previous similar message [ 227.733645] Lustre: lustre-MDT0001: new disk, initializing [ 227.811890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.845840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.860376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.101411] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.434795] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.080065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.512994] Lustre: lustre-OST0000: new disk, initializing [ 250.516633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.523742] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.597361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.917901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.930824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.049156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.041917] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.514507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.760486] Lustre: lustre-OST0001: new disk, initializing [ 273.768723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.779135] Lustre: 9478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.904715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.831851] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.695656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.704612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.771245] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.927312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.524901] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 305.937827] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.021803] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing yml_node [ 315.498218] Lustre: DEBUG MARKER: Client: 2.17.53.71 [ 318.350448] Lustre: DEBUG MARKER: MDS: 2.17.53.71 [ 320.998232] Lustre: DEBUG MARKER: OSS: 2.17.53.71 [ 323.044444] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 19:31:47 EDT 2026 [ 341.986721] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 343.808849] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.929243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.943428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.965081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.000318] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.005037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.020390] Lustre: Skipped 2 previous similar messages [ 364.036017] Lustre: Skipped 3 previous similar messages [ 367.022639] Lustre: server umount lustre-MDT0000 complete [ 371.620394] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220757 with bad export cookie 2752721892969211430 [ 371.623385] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.628327] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 374.240837] LustreError: 12420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.243577] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.254476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.268316] LustreError: 12420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 374.282212] Lustre: Skipped 1 previous similar message [ 376.302440] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.305019] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.313757] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.334136] Lustre: Skipped 2 previous similar messages [ 378.112501] Lustre: server umount lustre-MDT0001 complete [ 383.132506] Lustre: server umount lustre-OST0000 complete [ 387.613788] Lustre: server umount lustre-OST0001 complete [ 395.933811] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 407.909520] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 445.683815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.031486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 446.067909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 446.143760] Lustre: lustre-MDT0000: new disk, initializing [ 446.224360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.240369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.455769] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 461.246899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.422873] Lustre: 14998:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 461.516095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 461.520357] Lustre: Skipped 1 previous similar message [ 461.596333] Lustre: lustre-MDT0001: new disk, initializing [ 461.694341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.743195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.758367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 466.045341] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 473.968268] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 477.448486] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.486865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.782139] Lustre: lustre-OST0000: new disk, initializing [ 484.785988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.794959] Lustre: 16738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.881514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.189319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 486.211218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 486.408647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.370758] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 499.456296] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 504.690184] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.994181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 509.969365] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.229768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 517.091995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.093803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 517.121662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.125226] Lustre: Skipped 1 previous similar message [ 527.327422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.346502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.355447] Lustre: Skipped 3 previous similar messages [ 527.554666] Lustre: server umount lustre-OST0000 complete [ 532.453352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.464760] Lustre: Skipped 2 previous similar messages [ 538.607447] Lustre: server umount lustre-MDT0000 complete [ 542.702390] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.731582] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 542.911100] LustreError: 14988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220928 with bad export cookie 2752721892969212963 [ 542.931975] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.934690] LustreError: 14988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 543.574667] Lustre: server umount lustre-MDT0001 complete [ 550.654458] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:35:35 (1781220935) [ 571.775352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.312134] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.401384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.511869] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.808524] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 587.428072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.751879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 587.762051] Lustre: Skipped 4 previous similar messages [ 592.767697] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 599.536603] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 603.817789] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 606.017316] Lustre: Failing over lustre-MDT0000 [ 606.042032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.340806] Lustre: server umount lustre-MDT0000 complete [ 611.296979] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.314786] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 626.008605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.146673] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.500453] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 631.785623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 631.796989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 631.851480] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 631.997581] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 640.772413] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 647.139086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 647.140854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.161842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.161850] Lustre: Skipped 2 previous similar messages [ 647.201575] Lustre: Skipped 2 previous similar messages [ 651.844478] Lustre: server umount lustre-MDT0000 complete [ 652.264299] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.286677] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 656.203954] LustreError: 21791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221042 with bad export cookie 2752721892969214377 [ 656.212167] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.231352] LustreError: 21791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 656.571145] Lustre: server umount lustre-MDT0001 complete [ 666.175647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.877320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.883511] Lustre: Skipped 2 previous similar messages [ 671.194534] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 680.444814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.551912] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 692.450567] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 696.668948] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 698.749472] Lustre: Failing over lustre-MDT0000 [ 699.217914] Lustre: server umount lustre-MDT0000 complete [ 701.416252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.418304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.432350] LustreError: 22469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.432361] LustreError: 22469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 701.492432] Lustre: Skipped 1 previous similar message [ 717.279165] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781221087/real 1781221087] req@ffff9770bf81b840 x1867745077681664/t0(0) o400->MGC192.168.202.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781221103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 717.320532] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.280680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.220684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 733.156835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 733.173792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 733.179891] Lustre: Skipped 1 previous similar message [ 733.209243] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 733.757316] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 741.382966] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 747.490044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 747.493613] Lustre: Skipped 1 previous similar message [ 752.519505] Lustre: server umount lustre-MDT0000 complete [ 758.101500] LustreError: 22449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221143 with bad export cookie 2752721892969215315 [ 758.132678] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 759.163122] Lustre: server umount lustre-MDT0001 complete [ 771.993413] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:39:16 (1781221156) [ 792.015946] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 803.438385] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 848.110817] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 860.893389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.257820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.305604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 861.414924] Lustre: lustre-MDT0000: new disk, initializing [ 861.530973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.537903] Lustre: Skipped 2 previous similar messages [ 861.590179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 865.901328] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 877.141537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.282711] Lustre: 28458:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 877.675837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 882.875348] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 892.185701] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.448591] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 904.243448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.644909] Lustre: 30201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 905.856254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 905.921459] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 906.002597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 906.025840] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 911.845446] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 919.724956] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.395235] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 947.174357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.180355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.184173] Lustre: Skipped 2 previous similar messages [ 947.185678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.205941] Lustre: Skipped 3 previous similar messages [ 951.240732] Lustre: server umount lustre-OST0000 complete [ 961.744698] Lustre: server umount lustre-MDT0000 complete [ 962.544898] LustreError: 28468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.564432] LustreError: 28468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 966.179374] LustreError: 28450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221352 with bad export cookie 2752721892969215721 [ 966.181141] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.190790] LustreError: 28450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 966.718893] Lustre: server umount lustre-MDT0001 complete [ 979.340900] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 982.435650] Key type lgssc unregistered [ 982.714863] LNet: 32047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.727900] LNetError: 32047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.749667] LNet: Removed LNI 192.168.202.148@tcp [ 983.963212] Key type .llcrypt unregistered [ 983.966199] Key type ._llcrypt unregistered [ 1020.463168] Key type ._llcrypt registered [ 1020.472681] Key type .llcrypt registered [ 1020.755151] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1023.083635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.130218] alg: No test for adler32 (adler32-zlib) [ 1024.385978] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 1024.831207] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1026.575241] Key type lgssc registered [ 1027.810740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.474897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.505571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.981405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.945423] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1048.547314] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.576174] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1052.929832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.444355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.532923] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1063.435971] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1068.289826] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.982130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.847586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.848560] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1083.971457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1086.683554] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1093.045431] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1107.876567] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:44:52 (1781221492) [ 1109.843153] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1111.991881] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1120.028550] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:45:04 (1781221504) [ 1131.914703] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:45:15 (1781221515) [ 1133.716735] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1146.161720] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1146.167547] Lustre: Skipped 2 previous similar messages [ 1149.134557] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:45:32 (1781221532) [ 1150.905184] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1150.916571] Lustre: Skipped 2 previous similar messages [ 1160.629429] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1160.632463] Lustre: Skipped 4 previous similar messages [ 1177.226275] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:46:01 (1781221561) [ 1180.082977] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.048705] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1193.391922] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:46:18 (1781221578) [ 1195.406583] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1195.411726] Lustre: Skipped 6 previous similar messages [ 1209.944889] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:46:34 (1781221594) [ 1232.365455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.369824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.383818] LustreError: Skipped 1 previous similar message [ 1232.406339] Lustre: Skipped 1 previous similar message [ 1232.415917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.420677] Lustre: Skipped 1 previous similar message [ 1234.949347] Lustre: server umount lustre-OST0000 complete [ 1242.595082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.626717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.631774] Lustre: Skipped 1 previous similar message [ 1245.898087] Lustre: server umount lustre-MDT0000 complete [ 1247.712745] LustreError: 33445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.730826] LustreError: 33445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1249.958764] LustreError: 38149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221635 with bad export cookie 17779971562198795771 [ 1249.969092] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.971838] LustreError: 38149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1250.313409] Lustre: server umount lustre-MDT0001 complete [ 1261.759177] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1265.263119] Key type lgssc unregistered [ 1265.573076] LNet: 40961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.577467] LNetError: 40961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.597382] LNet: Removed LNI 192.168.202.148@tcp [ 1266.496423] Key type .llcrypt unregistered [ 1266.501321] Key type ._llcrypt unregistered [ 1285.966902] Key type ._llcrypt registered [ 1285.970433] Key type .llcrypt registered [ 1286.116551] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 1301.923506] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1303.189190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.242474] alg: No test for adler32 (adler32-zlib) [ 1304.268156] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 1304.490071] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1306.216463] Key type lgssc registered [ 1307.682761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.499776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.534114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.863960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1350.901812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1351.018461] Lustre: lustre-MDT0000: new disk, initializing [ 1351.091633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.125506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1356.102905] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1366.701789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.813900] Lustre: 44084:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1366.923539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1366.934536] Lustre: Skipped 1 previous similar message [ 1366.941683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1367.206425] Lustre: lustre-MDT0001: new disk, initializing [ 1367.440920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.525135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1367.547241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1372.545792] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1380.708474] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1384.222464] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1391.275780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.556759] Lustre: lustre-OST0000: new disk, initializing [ 1391.561853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1391.574518] Lustre: 45826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1391.723892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.844752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1396.851819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1396.965821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1397.544824] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1406.033367] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.711727] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1410.971747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1415.289295] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1415.502978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1417.696794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.698186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.707272] LustreError: Skipped 1 previous similar message [ 1417.727622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.823346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.826344] Lustre: Skipped 2 previous similar messages [ 1423.638775] Lustre: server umount lustre-OST0000 complete [ 1427.936340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.945869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.946666] Lustre: Skipped 2 previous similar messages [ 1427.960521] Lustre: Skipped 1 previous similar message [ 1433.058548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.067199] Lustre: Skipped 1 previous similar message [ 1433.865321] Lustre: server umount lustre-MDT0000 complete [ 1437.865130] LustreError: 44076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221823 with bad export cookie 17662428899809985444 [ 1437.875173] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.878155] LustreError: 44076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1438.634344] Lustre: server umount lustre-MDT0001 complete [ 1452.270274] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:50:35 (1781221835) [ 1458.952602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.536133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.718976] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1464.805105] LustreError: 48138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.837938] LustreError: 48138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1469.926437] LustreError: 48139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.325866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.830980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.922913] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1484.213561] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.677179] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1494.611974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.024251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.810890] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1510.356073] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.227653] LustreError: 50327:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1516.505439] Lustre: Modifying parameter general.timeout=1 in log params [ 1518.433582] Lustre: Modifying parameter general.timeout=3 in log params [ 1518.440177] Lustre: Skipped 1 previous similar message [ 1521.120888] Lustre: Modifying parameter general.timeout=6 in log params [ 1521.126556] Lustre: Skipped 2 previous similar messages [ 1525.845896] Lustre: Modifying parameter general.timeout=11 in log params [ 1525.852913] Lustre: Skipped 4 previous similar messages [ 1534.517758] Lustre: Modifying parameter general.timeout=20 in log params [ 1534.521683] Lustre: Skipped 8 previous similar messages [ 1550.605946] Lustre: Modifying parameter general.timeout=38 in log params [ 1550.612103] Lustre: Skipped 17 previous similar messages [ 1582.637187] Lustre: Modifying parameter general.timeout=69 in log params [ 1582.640883] Lustre: Skipped 30 previous similar messages [ 1648.892865] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:53:53 (1781222033) [ 1649.884496] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1649.888178] Lustre: Skipped 61 previous similar messages [ 1651.683568] Lustre: 57284:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1653.684304] Lustre: 57383:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1657.927378] Lustre: 57580:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1657.940462] Lustre: 57580:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1662.534471] Lustre: 57790:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1662.573166] Lustre: 57790:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1695.525202] Lustre: 59158:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1695.549667] LustreError: 59158:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1700.217245] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:54:44 (1781222084) [ 1715.179329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.184238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.188858] Lustre: Skipped 2 previous similar messages [ 1715.192082] Lustre: Skipped 1 previous similar message [ 1719.376851] Lustre: server umount lustre-MDT0000 complete [ 1720.293235] LustreError: 48138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.323746] LustreError: 48138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1724.313409] LustreError: 48120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222110 with bad export cookie 17662428899809986326 [ 1724.322849] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1724.337336] LustreError: 48120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1724.873643] Lustre: server umount lustre-MDT0001 complete [ 1740.572092] Lustre: server umount lustre-OST0000 complete [ 1769.256845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.500437] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1769.712427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.492707] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1782.825037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.885885] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1782.904858] Lustre: 61694:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1787.289647] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1793.657679] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1797.699692] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1806.123976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.273526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1806.706694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.709365] Lustre: Skipped 1 previous similar message [ 1812.656619] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1818.723482] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.508560] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1820.521786] Lustre: Skipped 22 previous similar messages [ 1820.563411] Lustre: 63922:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1828.320942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.326778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.337469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.342460] Lustre: Skipped 1 previous similar message [ 1834.572050] Lustre: server umount lustre-OST0000 complete [ 1839.584696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1839.589109] LustreError: Skipped 1 previous similar message [ 1839.591573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.596769] Lustre: Skipped 1 previous similar message [ 1846.280836] Lustre: server umount lustre-MDT0000 complete [ 1849.313221] LustreError: 61699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.329745] LustreError: 61699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1851.624562] LustreError: 61686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222237 with bad export cookie 17662428899809989483 [ 1851.646130] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.648029] LustreError: 61686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1852.778383] Lustre: server umount lustre-MDT0001 complete [ 1865.244262] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1868.321667] Key type lgssc unregistered [ 1868.750397] LNet: 65114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.760655] LNetError: 65114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.779669] LNet: Removed LNI 192.168.202.148@tcp [ 1870.073289] Key type .llcrypt unregistered [ 1870.076430] Key type ._llcrypt unregistered [ 1900.063782] Key type ._llcrypt registered [ 1900.065566] Key type .llcrypt registered [ 1900.203882] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1902.132304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.161966] alg: No test for adler32 (adler32-zlib) [ 1903.315740] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 1903.686046] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1905.447394] Key type lgssc registered [ 1906.873299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1916.460651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1916.513755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.252639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.707151] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1928.673685] LustreError: 66512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.708160] LustreError: 66512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1933.160569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.848923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1939.001437] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1946.030131] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1950.592930] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1958.626138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.184296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.265797] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1972.982480] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1984.095991] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1994.213442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.217144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.247129] Lustre: Skipped 1 previous similar message [ 1994.269044] Lustre: Skipped 2 previous similar messages [ 1999.330212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.335748] Lustre: Skipped 2 previous similar messages [ 2000.490185] Lustre: server umount lustre-MDT0000 complete [ 2004.361810] LustreError: 66493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222390 with bad export cookie 5607496711320138353 [ 2004.365973] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.372489] LustreError: 66493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2004.455593] LustreError: 66507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.456169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.504780] Lustre: Skipped 1 previous similar message [ 2004.515850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2004.707231] Lustre: server umount lustre-MDT0001 complete [ 2010.829674] Lustre: server umount lustre-OST0000 complete [ 2042.483715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.787682] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2042.823075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2043.063891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.979104] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2061.386721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.517388] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2061.560927] Lustre: 71053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2062.021639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.769744] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2073.276544] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.110280] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2085.957959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2086.223347] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2086.753312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.823571] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2102.541226] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2104.884914] Lustre: 73302:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2106.559927] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2116.576937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.579574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.596037] LustreError: Skipped 1 previous similar message [ 2116.597456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.641287] Lustre: Skipped 1 previous similar message [ 2120.162232] Lustre: server umount lustre-OST0000 complete [ 2126.828231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.844948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.856966] Lustre: Skipped 1 previous similar message [ 2126.862133] Lustre: Skipped 1 previous similar message [ 2131.237619] Lustre: server umount lustre-MDT0000 complete [ 2131.937614] LustreError: 72718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.969295] LustreError: 72718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2136.270162] LustreError: 71044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222522 with bad export cookie 5607496711320139200 [ 2136.272696] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.288591] LustreError: 71044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2137.036887] Lustre: server umount lustre-MDT0001 complete [ 2149.530973] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2152.768644] Key type lgssc unregistered [ 2153.118411] LNet: 74502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.126519] LNetError: 74502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.148771] LNet: Removed LNI 192.168.202.148@tcp [ 2154.023696] Key type .llcrypt unregistered [ 2154.025887] Key type ._llcrypt unregistered [ 2181.969402] Key type ._llcrypt registered [ 2181.973452] Key type .llcrypt registered [ 2182.142792] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2183.587288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.611270] alg: No test for adler32 (adler32-zlib) [ 2185.123071] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 2185.467688] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2187.223204] Key type lgssc registered [ 2188.395642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.337574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2196.350534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.149401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.252732] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2207.726178] LustreError: 75900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.757184] LustreError: 75900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2212.842261] LustreError: 75901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.404055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.230912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.220553] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2232.359248] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2237.064719] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2244.654285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2245.135989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.399828] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2258.779491] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2274.594101] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2275.761062] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2276.797613] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.088552] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.092572] Lustre: Skipped 1 previous similar message [ 2284.068429] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2284.077927] Lustre: Skipped 3 previous similar messages [ 2293.492080] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2293.509287] Lustre: Skipped 6 previous similar messages [ 2309.688680] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2309.693978] Lustre: Skipped 13 previous similar messages [ 2342.697357] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2342.703692] Lustre: Skipped 31 previous similar messages [ 2393.551141] Lustre: 83692:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2397.171593] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:06:21 (1781222781) [ 2399.016520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2400.975586] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:06:25 (1781222785) [ 2405.859903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.875939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.880513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.957144] Lustre: Skipped 1 previous similar message [ 2410.978765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.984283] Lustre: Skipped 2 previous similar messages [ 2411.878701] Lustre: server umount lustre-OST0000 complete [ 2416.610311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2416.621326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.640437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.217684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.222298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.267504] Lustre: Skipped 1 previous similar message [ 2422.643229] Lustre: server umount lustre-MDT0000 complete [ 2426.339780] LustreError: 76813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.373211] LustreError: 76813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2426.954032] LustreError: 75882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222812 with bad export cookie 3106012169901026665 [ 2426.969237] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.977675] LustreError: 75882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2427.393612] Lustre: server umount lustre-MDT0001 complete [ 2439.570378] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2443.251842] Key type lgssc unregistered [ 2443.601180] LNet: 84929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.609032] LNetError: 84929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.645632] LNet: Removed LNI 192.168.202.148@tcp [ 2444.518432] Key type .llcrypt unregistered [ 2444.520156] Key type ._llcrypt unregistered [ 2449.681913] Key type ._llcrypt registered [ 2449.683575] Key type .llcrypt registered [ 2449.841108] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2455.118941] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules [ 2456.532918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.571641] alg: No test for adler32 (adler32-zlib) [ 2457.788530] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 2458.300968] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2460.159657] Key type lgssc registered [ 2462.165737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2464.381848] LustreError: 86145:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2486.606711] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2486.711614] LustreError: 86145:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2499.256627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2499.283888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.311797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.948613] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2510.831234] LustreError: 87151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.861081] LustreError: 87151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2515.936976] LustreError: 87152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.452990] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:08:22 (1781222902) [ 2521.057669] LustreError: 87151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.628439] LustreError: 87183:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2524.645911] LustreError: 87183:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2524.654138] LustreError: 87183:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2525.159883] Lustre: server umount lustre-MDT0000 complete [ 2540.483332] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2544.087392] Key type lgssc unregistered [ 2544.371645] LNet: 88645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.381452] LNetError: 88645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.400378] LNet: Removed LNI 192.168.202.148@tcp [ 2545.304095] Key type .llcrypt unregistered [ 2545.305636] Key type ._llcrypt unregistered [ 2567.135345] Key type ._llcrypt registered [ 2567.138906] Key type .llcrypt registered [ 2567.226595] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2568.244229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2568.284737] alg: No test for adler32 (adler32-zlib) [ 2569.559716] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 2569.871696] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2571.591850] Key type lgssc registered [ 2573.186697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.636024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.677875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.487655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.250783] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2593.770895] LustreError: 89846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.791662] LustreError: 89846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2598.889167] LustreError: 89847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.033377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.151995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.254620] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2616.884649] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2622.722979] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2632.065052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.286868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.274529] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2650.585348] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2707.262430] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:11:31 (1781223091) [ 2709.961907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2713.041927] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:11:36 (1781223096) [ 2720.224515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.233251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.257719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.747672] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.782249] Lustre: Skipped 1 previous similar message [ 2720.792297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.804084] Lustre: Skipped 1 previous similar message [ 2723.841767] Lustre: server umount lustre-MDT0000 complete [ 2725.866623] LustreError: 90562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.902070] LustreError: 90562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2728.608884] LustreError: 89826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223114 with bad export cookie 9913707350929522584 [ 2728.631270] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2728.641265] LustreError: 89826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2729.272346] Lustre: server umount lustre-MDT0001 complete [ 2744.910653] Lustre: server umount lustre-OST0000 complete [ 2759.242388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.866957] LustreError: 93807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.160365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.280169] LustreError: 93808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.294374] LustreError: 93808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2765.980985] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2774.498919] LustreError: 93807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.518137] LustreError: 93807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2776.002227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.999610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.138348] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2789.109945] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2793.783482] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2809.199204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.384443] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2809.390156] LustreError: 93796:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2809.434174] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2809.442940] LustreError: 95669:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2809.449734] LustreError: 95669:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2809.454291] LustreError: 95669:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2809.523706] Lustre: server umount lustre-OST0000 complete [ 2809.526702] LustreError: 95669:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2818.343521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.512365] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2818.517399] LustreError: 93796:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2818.524531] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2818.534338] LustreError: 96130:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2818.539398] LustreError: 96130:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2818.543638] LustreError: 96130:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2818.596808] Lustre: server umount lustre-OST0000 complete [ 2818.599523] LustreError: 96130:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2828.149555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.515835] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2828.527758] Lustre: Found index 0 for lustre-OST0000, updating log [ 2828.541798] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2828.635989] Lustre: lustre-OST0000: new disk, initializing [ 2828.644611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2828.656810] Lustre: 96663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2829.083716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.660505] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2840.790951] Lustre: Failing over lustre-OST0000 [ 2840.919977] Lustre: server umount lustre-OST0000 complete [ 2845.666318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.703180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.712060] Lustre: Skipped 1 previous similar message [ 2851.812416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.820126] Lustre: Skipped 3 previous similar messages [ 2855.022960] Lustre: server umount lustre-MDT0000 complete [ 2856.929501] LustreError: 93807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.969785] LustreError: 93807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2860.648571] LustreError: 93790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223246 with bad export cookie 9913707350929525328 [ 2860.659423] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.683333] LustreError: 93790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2861.106537] Lustre: server umount lustre-MDT0001 complete [ 2875.125740] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:14:18 (1781223258) [ 2895.958772] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2908.382301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.010622] LustreError: 98978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.177288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.284675] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2926.610529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.338598] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2936.689660] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2945.053782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2945.528600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.532317] Lustre: Skipped 1 previous similar message [ 2945.549028] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2951.659826] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2951.669730] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2951.691191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2953.013201] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2953.710970] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:07 [ 2953.741239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2953.757407] Lustre: lustre-OST0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 2953.760956] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2953.788166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2964.355284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.501674] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2964.758205] Lustre: lustre-OST0001: new disk, initializing [ 2964.778672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2964.798721] Lustre: 101250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2972.335116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2972.362057] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2972.574101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2973.506793] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2984.250970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2995.284267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3003.367452] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.373822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3003.412360] Lustre: Skipped 2 previous similar messages [ 3003.437235] Lustre: Skipped 3 previous similar messages [ 3006.934886] Lustre: server umount lustre-MDT0001 complete [ 3008.485122] LustreError: 98975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.524916] LustreError: 98975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3018.911329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.009740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3019.012064] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3019.058637] Lustre: 98967:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3020.088242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.098630] Lustre: Skipped 1 previous similar message [ 3020.347510] LustreError: 102754:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3020.360835] LustreError: 102754:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3020.367692] LustreError: 98970:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3020.526459] LustreError: 102769:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -17 [ 3020.537141] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3020.537141] [ 3025.402637] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3025.500321] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3037.831717] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:17:02 (1781223422) [ 3064.636219] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3089.567992] Lustre: 104850:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3118.978593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3124.983142] Lustre: 105985:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3219.395214] Lustre: server umount lustre-MDT0000 complete [ 3219.430157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.435182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.455764] Lustre: Skipped 1 previous similar message [ 3219.471844] LustreError: 98979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.489931] LustreError: 98979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3223.907430] LustreError: 101953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223609 with bad export cookie 9913707350929526259 [ 3223.909194] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.918190] LustreError: 101953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.407318] Lustre: server umount lustre-MDT0001 complete [ 3239.639429] Lustre: server umount lustre-OST0000 complete [ 3240.543412] Lustre: 89158:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781223610/real 1781223610] req@ffff976f922b8f00 x1867747641643904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781223626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3240.592594] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.624494] Lustre: Skipped 3 previous similar messages [ 3243.231388] Lustre: 89157:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781223613/real 1781223613] req@ffff9770b10f6940 x1867747641644160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781223629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3244.500243] Lustre: server umount lustre-OST0001 complete [ 3251.074504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3263.334694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3274.558686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3286.851703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3310.539379] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3324.156142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.203297] Lustre: lustre-MDT0000: reset Object Index mappings [ 3325.165339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.561645] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3341.911968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.951340] Lustre: lustre-MDT0001: reset Object Index mappings [ 3348.866897] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3352.844299] Lustre: 111035:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3361.487674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.811979] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3369.890739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3371.020184] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3383.319050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.959524] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3389.963935] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3393.239604] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3403.431921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3407.580383] Lustre: 112880:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3468.704313] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:24:12 (1781223852) [ 3496.416709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3496.431813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.456186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3499.520439] Lustre: server umount lustre-MDT0000 complete [ 3502.567102] LustreError: 110652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.596402] LustreError: 110652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 3504.447853] LustreError: 109901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223890 with bad export cookie 9913707350929573432 [ 3504.457777] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.482877] LustreError: 109901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3511.044396] Lustre: server umount lustre-MDT0001 complete [ 3515.983219] LustreError: 113637:0:(ldlm_resource.c:1180:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x132:0x0].0x0 (ffff9770921b3c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3522.250949] Lustre: server umount lustre-OST0000 complete [ 3532.952161] Lustre: server umount lustre-OST0001 complete [ 3540.603666] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 3552.976956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3596.189954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.434558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3596.459948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3596.543283] Lustre: lustre-MDT0000: new disk, initializing [ 3596.624627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.629237] Lustre: Skipped 3 previous similar messages [ 3596.644589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3601.526826] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3608.951875] Lustre: server umount lustre-MDT0000 complete [ 3621.000834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.325988] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3621.342861] Lustre: 117098:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3621.356066] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3627.478023] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3633.575810] Lustre: server umount lustre-MDT0000 complete [ 3643.666843] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:27:08 (1781224028) [ 3645.702994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3648.064604] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:27:12 (1781224032) [ 3670.567887] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3681.903476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.356559] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3697.792613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.989380] Lustre: 119134:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3698.063692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3698.067622] Lustre: Skipped 1 previous similar message [ 3698.201735] Lustre: lustre-MDT0001: new disk, initializing [ 3698.469864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3698.473180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3704.214199] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3710.359949] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3718.987164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.399446] Lustre: lustre-OST0000: new disk, initializing [ 3719.407637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3719.416377] Lustre: 120678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3721.019100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3721.041664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3721.203594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3727.238938] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3738.769439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.958355] Lustre: lustre-OST0001: new disk, initializing [ 3738.962343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3738.966458] Lustre: 121532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.498536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3740.506266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3740.621900] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3745.133130] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3756.118041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3760.612256] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4177.697041] Lustre: 127932:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4177.702549] Lustre: 127932:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4177.706795] Lustre: 127932:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4177.712073] Lustre: 127932:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4177.717459] Lustre: 127932:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4177.722421] Lustre: 127932:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4492.877533] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:41:16 (1781224876) [ 4512.736373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.751472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.763154] Lustre: Skipped 5 previous similar messages [ 4512.777510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.789424] Lustre: Skipped 8 previous similar messages [ 4518.285501] Lustre: server umount lustre-MDT0000 complete [ 4518.891970] LustreError: 122319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.908979] LustreError: 122319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4522.879226] LustreError: 119125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224908 with bad export cookie 9913707350929643089 [ 4522.881574] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.892773] LustreError: 119125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4524.000841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.006825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4524.020030] Lustre: Skipped 4 previous similar messages [ 4524.030507] Lustre: Skipped 5 previous similar messages [ 4528.033552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4528.046279] Lustre: Skipped 1 previous similar message [ 4537.311751] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4537.315280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4537.351041] Lustre: Skipped 3 previous similar messages [ 4538.151801] Lustre: server umount lustre-MDT0001 complete [ 4549.794973] Lustre: server umount lustre-OST0000 complete [ 4560.231385] Lustre: server umount lustre-OST0001 complete [ 4570.515538] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 4589.293454] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4637.700548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.878209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4637.893471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4637.941193] Lustre: lustre-MDT0000: new disk, initializing [ 4638.039847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.047576] Lustre: Skipped 5 previous similar messages [ 4638.067491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4643.799820] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4657.611875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.707212] Lustre: 130991:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4657.752558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4657.760770] Lustre: Skipped 1 previous similar message [ 4657.829449] Lustre: lustre-MDT0001: new disk, initializing [ 4657.902913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4657.906580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4663.964260] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4673.633597] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4679.214568] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4689.224715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4689.813336] Lustre: lustre-OST0000: new disk, initializing [ 4689.820754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4689.827534] Lustre: 132738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4691.487547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4691.500360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4691.709621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4699.537913] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4709.190165] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4720.939253] Lustre: lustre-MDD0000: changelog on [ 4724.575440] Lustre: lustre-MDD0001: changelog on [ 6706.219230] Lustre: lustre-MDD0001: changelog off [ 6709.167948] Lustre: 130999:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6709.910358] Lustre: lustre-MDD0000: changelog off [ 6719.462624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6719.463203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6719.475076] LustreError: Skipped 1 previous similar message [ 6719.476227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6719.507353] Lustre: Skipped 1 previous similar message [ 6724.037624] Lustre: server umount lustre-OST0000 complete [ 6729.193728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.212749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6729.218416] Lustre: Skipped 2 previous similar messages [ 6734.306521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6735.773474] Lustre: server umount lustre-MDT0000 complete [ 6739.428044] LustreError: 131004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.451795] LustreError: 131004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6740.211245] LustreError: 130986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227126 with bad export cookie 9913707350930465372 [ 6740.230426] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6740.237552] LustreError: 130986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6740.776645] Lustre: server umount lustre-MDT0001 complete [ 6756.163752] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6759.957687] Key type lgssc unregistered [ 6760.374537] LNet: 141771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6760.379644] LNetError: 141771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6760.416744] LNet: Removed LNI 192.168.202.148@tcp [ 6761.754165] Key type .llcrypt unregistered [ 6761.759286] Key type ._llcrypt unregistered [ 6785.326292] Key type ._llcrypt registered [ 6785.328513] Key type .llcrypt registered [ 6785.452828] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 6803.275377] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6804.966525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6805.349925] alg: No test for adler32 (adler32-zlib) [ 6806.483141] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 6806.796443] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6808.487189] Key type lgssc registered [ 6810.021363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6854.634163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6854.691801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.110678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6856.171664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6856.291608] Lustre: lustre-MDT0000: new disk, initializing [ 6856.432208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.458876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6862.104801] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6874.043235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.152713] Lustre: 144942:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6874.195523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6874.201867] Lustre: Skipped 1 previous similar message [ 6874.328945] Lustre: lustre-MDT0001: new disk, initializing [ 6874.483606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6874.517034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6874.522175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6880.283699] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6888.991478] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6893.897709] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6902.699705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.079571] Lustre: lustre-OST0000: new disk, initializing [ 6903.098399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6903.110057] Lustre: 146683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6903.254537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6909.551723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6909.561302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6909.732444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6909.883717] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6918.453293] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6925.325630] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_module obdecho/obdecho [ 6927.535747] LustreError: 147558:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6927.542604] LustreError: 147558:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6929.617519] LustreError: 147654:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6945.766624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.772328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6945.786168] Lustre: Skipped 1 previous similar message [ 6945.805250] Lustre: Skipped 2 previous similar messages [ 6948.094656] Lustre: server umount lustre-MDT0000 complete [ 6952.172198] LustreError: 144933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227338 with bad export cookie 6921111117887873856 [ 6952.179927] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6952.194512] LustreError: 144933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6952.588613] Lustre: server umount lustre-MDT0001 complete [ 6967.152466] Lustre: server umount lustre-OST0000 complete [ 6990.038931] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 7004.624892] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7052.247900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.684490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7052.756926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7052.887802] Lustre: lustre-MDT0000: new disk, initializing [ 7053.008552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.031600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7058.206244] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7070.573025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.762954] Lustre: 151336:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7070.874505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7070.880638] Lustre: Skipped 1 previous similar message [ 7070.985456] Lustre: lustre-MDT0001: new disk, initializing [ 7071.064361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.110667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7071.130036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7076.249263] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7084.972274] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7089.149973] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7096.204976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.552981] Lustre: lustre-OST0000: new disk, initializing [ 7096.556532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7096.566988] Lustre: 153080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7096.698203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.549584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7103.562276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7103.683702] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7103.741615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7112.245384] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7117.676827] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7117.999825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7123.130181] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7123.337634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7129.571756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7129.578869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.592984] LustreError: Skipped 1 previous similar message [ 7129.600109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.615683] Lustre: Skipped 2 previous similar messages [ 7129.642834] Lustre: Skipped 1 previous similar message [ 7134.690214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.713725] Lustre: Skipped 1 previous similar message [ 7139.823723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.836487] Lustre: Skipped 1 previous similar message [ 7140.319157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7140.644783] Lustre: server umount lustre-OST0000 complete [ 7144.933861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.952809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.959707] Lustre: Skipped 1 previous similar message [ 7150.909957] Lustre: server umount lustre-MDT0000 complete [ 7155.024572] LustreError: 153079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227540 with bad export cookie 6921111117887874808 [ 7155.025807] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7155.034399] LustreError: 153079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7155.340623] Lustre: server umount lustre-MDT0001 complete [ 7159.738970] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:25:44 (1781227544) [ 7161.263126] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7163.462287] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:25:48 (1781227548) [ 7171.934235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.363574] LustreError: 155348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.385953] LustreError: 155348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7172.442610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.302624] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7177.699340] LustreError: 155349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.819811] LustreError: 155348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.376915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.652123] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7197.838627] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7202.087924] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7209.193571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.706607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.711911] Lustre: Skipped 1 previous similar message [ 7216.478249] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7222.282126] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7227.872405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7227.877614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.877722] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 7227.883976] Lustre: Skipped 1 previous similar message [ 7227.890601] Lustre: Skipped 3 previous similar messages [ 7230.086527] Lustre: server umount lustre-MDT0000 complete [ 7232.991315] LustreError: 155343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.577913] LustreError: 155329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227619 with bad export cookie 6921111117887875690 [ 7233.594932] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.599304] LustreError: 155329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7238.111659] LustreError: 155344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.148824] LustreError: 155344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7238.644168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.659789] Lustre: Skipped 2 previous similar messages [ 7240.021799] Lustre: server umount lustre-MDT0001 complete [ 7247.159105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7260.727496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.031448] LustreError: 158849:0:(llog.c:1646:llog_backup()) MGC192.168.202.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7261.044041] Lustre: 158849:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7267.877104] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.897835] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7284.704607] LustreError: 158856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7284.716537] LustreError: 158856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 7319.003872] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.030539] LustreError: 158854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 7322.080642] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7322.096251] Lustre: 158849:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7363.077320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7368.165130] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7368.772433] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7378.718043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.771518] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7385.090362] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7391.182704] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7393.226881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7397.621660] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7399.552658] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7420.907897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.908442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7420.929061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7420.929071] Lustre: Skipped 5 previous similar messages [ 7420.945835] Lustre: Skipped 2 previous similar messages [ 7432.159174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7432.565393] Lustre: server umount lustre-MDT0000 complete [ 7436.266454] LustreError: 158855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.276920] LustreError: 158855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 7436.959510] LustreError: 155330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227822 with bad export cookie 6921111117887882151 [ 7436.962634] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.969728] LustreError: 155330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7437.412539] Lustre: server umount lustre-MDT0001 complete [ 7452.277952] Lustre: server umount lustre-OST0000 complete [ 7476.609875] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 7486.923957] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7522.940814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.296893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7523.327975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7523.408466] Lustre: lustre-MDT0000: new disk, initializing [ 7523.510423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.514444] Lustre: Skipped 1 previous similar message [ 7523.528523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7527.793659] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7538.480473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.620784] Lustre: 163698:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7538.978899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7544.093309] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7552.094811] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7556.373780] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7562.618953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.836591] Lustre: lustre-OST0000: new disk, initializing [ 7562.839072] Lustre: Skipped 1 previous similar message [ 7562.842665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7562.846543] Lustre: Skipped 2 previous similar messages [ 7562.851312] Lustre: 165441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7569.177936] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7571.030526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7571.035569] Lustre: Skipped 1 previous similar message [ 7571.044789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7571.104607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7576.277455] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7581.323227] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7581.631560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7586.688041] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7586.917985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7591.905269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.913764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.925275] LustreError: Skipped 1 previous similar message [ 7591.931272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.931280] Lustre: Skipped 8 previous similar messages [ 7591.972435] Lustre: Skipped 1 previous similar message [ 7595.161392] Lustre: server umount lustre-OST0000 complete [ 7600.611496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7605.598936] Lustre: server umount lustre-MDT0000 complete [ 7607.265904] LustreError: 163705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.304182] LustreError: 163705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7610.143053] LustreError: 163691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227996 with bad export cookie 6921111117887898979 [ 7610.154573] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.166249] LustreError: 163691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7610.707908] Lustre: server umount lustre-MDT0001 complete [ 7621.556226] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:33:26 (1781228006) [ 7629.506852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.694640] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7646.280262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.528753] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7656.765728] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7660.600876] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7668.959512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7676.779566] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7683.511919] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7695.833814] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:34:39 (1781228079) [ 7702.520264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7702.530910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.571170] Lustre: Skipped 2 previous similar messages [ 7702.603173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7702.608139] Lustre: Skipped 4 previous similar messages [ 7707.204922] Lustre: server umount lustre-OST0000 complete [ 7718.227582] Lustre: server umount lustre-MDT0000 complete [ 7723.274214] LustreError: 167686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228109 with bad export cookie 6921111117887899868 [ 7723.274892] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.285341] LustreError: 167686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7723.890432] Lustre: server umount lustre-MDT0001 complete [ 7735.791300] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7737.939355] Key type lgssc unregistered [ 7738.224955] LNet: 171081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7738.236944] LNetError: 171081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7738.254776] LNet: Removed LNI 192.168.202.148@tcp [ 7739.338092] Key type .llcrypt unregistered [ 7739.339400] Key type ._llcrypt unregistered [ 7760.838668] Key type ._llcrypt registered [ 7760.842968] Key type .llcrypt registered [ 7760.960265] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7761.877065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7761.913340] alg: No test for adler32 (adler32-zlib) [ 7763.276768] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 7763.582897] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7765.287177] Key type lgssc registered [ 7766.429612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7774.861930] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7774.893968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7790.431883] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7795.551232] LustreError: 172308:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 7800.671977] LustreError: 172308:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7800.690455] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7800.698172] LustreError: MGC192.168.202.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7800.716931] LustreError: 172308:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7800.730053] LustreError: 172308:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7800.740071] LustreError: 172308:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7800.865791] Lustre: server umount lustre-OST0000 complete [ 7800.868304] LustreError: 172308:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7811.352500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.923334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.341541] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7822.315749] LustreError: 172809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.333766] LustreError: 172809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7827.428209] LustreError: 172810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.916691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.387635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7832.435860] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7837.969449] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7842.311820] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7850.297051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.583117] LustreError: 174401:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7850.592614] Lustre: 174401:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7850.795574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.224949] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7864.654512] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7868.386946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.396217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.411668] Lustre: Skipped 2 previous similar messages [ 7868.427400] Lustre: Skipped 2 previous similar messages [ 7873.521757] LustreError: 173521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.558589] LustreError: 173521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7875.545477] Lustre: server umount lustre-MDT0000 complete [ 7878.633348] LustreError: 172806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.657775] LustreError: 172806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7879.520405] LustreError: 172788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228265 with bad export cookie 7200262420921753603 [ 7879.532296] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7879.550623] LustreError: 172788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7879.929700] Lustre: server umount lustre-MDT0001 complete [ 7895.270346] Lustre: server umount lustre-OST0000 complete [ 7906.025857] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 7917.400323] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7959.805403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.195503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7960.223571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7960.377793] Lustre: lustre-MDT0000: new disk, initializing [ 7960.509950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.537726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7963.795408] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7976.182155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.428346] Lustre: 177778:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7976.586241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7976.591283] Lustre: Skipped 1 previous similar message [ 7976.737733] Lustre: lustre-MDT0001: new disk, initializing [ 7976.886090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7976.981302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7977.001190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7980.295962] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7988.903507] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7993.423815] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8001.255382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.603743] Lustre: lustre-OST0000: new disk, initializing [ 8001.608227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8001.613678] Lustre: 179518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8001.721357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.423858] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8009.329917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8009.346489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8009.497271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8016.430962] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8021.499509] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8021.801905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8026.382763] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8026.686841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8030.180044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.201130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.208570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.214975] LustreError: Skipped 1 previous similar message [ 8030.226508] Lustre: Skipped 1 previous similar message [ 8034.860370] Lustre: server umount lustre-OST0000 complete [ 8040.425858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.428689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.441656] Lustre: Skipped 2 previous similar messages [ 8040.455274] Lustre: Skipped 1 previous similar message [ 8045.465256] Lustre: server umount lustre-MDT0000 complete [ 8045.541033] LustreError: 177790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.571820] LustreError: 177790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8049.270639] LustreError: 178555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228435 with bad export cookie 7200262420921754331 [ 8049.271947] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8049.277393] LustreError: 178555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8049.694132] Lustre: server umount lustre-MDT0001 complete [ 8059.686685] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:40:44 (1781228444) [ 8079.853976] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8083.599811] Key type lgssc unregistered [ 8084.166566] LNet: 182300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8084.173369] LNetError: 182300:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8084.199876] LNet: Removed LNI 192.168.202.148@tcp [ 8085.365137] Key type .llcrypt unregistered [ 8085.368705] Key type ._llcrypt unregistered [ 8104.706956] Key type ._llcrypt registered [ 8104.713985] Key type .llcrypt registered [ 8104.836798] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8105.917784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8106.073935] alg: No test for adler32 (adler32-zlib) [ 8107.250180] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 8107.555328] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8109.287189] Key type lgssc registered [ 8110.700361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.573717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.604500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.186363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.046726] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8132.585158] LustreError: 183505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.609240] LustreError: 183505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8137.710612] LustreError: 183505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.739117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.311698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.044245] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8151.702585] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.516227] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8163.512512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.859438] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8163.863971] LustreError: 185145:0:(llog.c:1687:llog_backup()) MGC192.168.202.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8163.869967] Lustre: 185145:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8164.025664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.299907] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8176.643090] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8180.197743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8180.210853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.226541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.785290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.792972] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.801647] Lustre: Skipped 1 previous similar message [ 8183.846178] Lustre: Skipped 1 previous similar message [ 8185.242080] Lustre: server umount lustre-MDT0000 complete [ 8188.917128] LustreError: 183506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.943152] LustreError: 183506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8189.796647] LustreError: 183485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228575 with bad export cookie 11156787282826420529 [ 8189.821222] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8190.091770] Lustre: server umount lustre-MDT0001 complete [ 8205.666493] Lustre: server umount lustre-OST0000 complete [ 8216.609213] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8229.104160] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8266.199355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.462576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8266.490142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8266.569438] Lustre: lustre-MDT0000: new disk, initializing [ 8266.717751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.741047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8271.891885] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8282.714316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.900289] Lustre: 188521:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8282.953452] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8282.956905] Lustre: Skipped 1 previous similar message [ 8282.960446] LustreError: 189288:0:(llog.c:1687:llog_backup()) MGC192.168.202.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8282.968143] LustreError: 189288:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8282.974327] Lustre: 189288:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8283.005426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8283.015259] Lustre: Skipped 1 previous similar message [ 8283.103351] Lustre: lustre-MDT0001: new disk, initializing [ 8283.196832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8283.237092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8283.249271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8287.749339] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8296.309962] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8300.512287] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8308.180146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.487651] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8308.490799] Lustre: Skipped 1 previous similar message [ 8308.495990] LustreError: 190240:0:(llog.c:1687:llog_backup()) MGC192.168.202.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8308.503506] LustreError: 190240:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8308.516174] Lustre: 190240:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8308.525830] Lustre: 190240:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8308.556409] Lustre: lustre-OST0000: new disk, initializing [ 8308.559906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8308.565605] Lustre: 190264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8308.663610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8313.969097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8313.986960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8314.168620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8316.624245] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8324.715986] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8329.435356] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.750399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.854285] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.245331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8339.954971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8339.957730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.976989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8339.977012] Lustre: Skipped 1 previous similar message [ 8340.031547] Lustre: Skipped 1 previous similar message [ 8344.154165] Lustre: server umount lustre-OST0000 complete [ 8349.665085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8349.679949] LustreError: Skipped 1 previous similar message [ 8349.693363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.707511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.710276] Lustre: Skipped 1 previous similar message [ 8355.298789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.323179] Lustre: Skipped 3 previous similar messages [ 8355.452545] Lustre: server umount lustre-MDT0000 complete [ 8359.394472] LustreError: 188535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.405889] LustreError: 188535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8359.900596] LustreError: 188514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228745 with bad export cookie 11156787282826421250 [ 8359.906848] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.916116] LustreError: 188514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8360.330222] Lustre: server umount lustre-MDT0001 complete [ 8368.604876] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:45:53 (1781228753) [ 8383.649978] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8394.779467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.119167] LustreError: 193227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.206942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.781558] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8405.475709] LustreError: 193228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.495075] LustreError: 193228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8408.556442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.584358] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8415.641924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8423.889111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.185295] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8444.628362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8444.866341] Lustre: lustre-OST0001: new disk, initializing [ 8444.870324] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8444.877806] Lustre: 195485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8445.001734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8445.009575] Lustre: Skipped 2 previous similar messages [ 8451.788828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8451.807705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8451.931202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8452.527702] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8466.933824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8474.963769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8477.030716] Lustre: DEBUG MARKER: ADD OST3 [ 8482.328733] loop: module loaded [ 8482.409681] loop0: detected capacity change from 0 to 204800000 [ 8482.874411] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8482.891921] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8482.917620] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8490.104069] loop0: detected capacity change from 0 to 204800000 [ 8490.522889] Lustre: DEBUG MARKER: STOP OST3 [ 8497.471090] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8497.749329] Lustre: lustre-OST0003: new disk, initializing [ 8497.754830] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8497.770833] Lustre: 197505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8499.778787] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8499.935811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8499.950587] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8500.014200] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8501.650601] Lustre: Failing over lustre-OST0003 [ 8501.799532] LustreError: 197661:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8501.871475] Lustre: server umount lustre-OST0003 complete [ 8504.850873] LustreError: 194682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.322887] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8505.333242] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.348684] Lustre: Skipped 1 previous similar message [ 8505.437722] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8514.858611] Lustre: DEBUG MARKER: START OST3 again [ 8522.497889] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8523.251544] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8523.257966] Lustre: Skipped 1 previous similar message [ 8523.270698] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8524.505717] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8525.120104] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8525.121056] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8532.187781] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8556.435380] Lustre: Failing over lustre-OST0003 [ 8556.680546] Lustre: server umount lustre-OST0003 complete [ 8559.098222] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.129281] LustreError: 194993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.155209] LustreError: 194993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8578.031140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8578.041180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8578.055303] Lustre: Skipped 1 previous similar message [ 8578.062235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8583.812674] Lustre: server umount lustre-MDT0000 complete [ 8588.342775] LustreError: 193207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228974 with bad export cookie 11156787282826422132 [ 8588.347661] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8588.360527] LustreError: 193207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8595.294889] Lustre: server umount lustre-MDT0001 complete [ 8599.640912] Lustre: server umount lustre-OST0000 complete [ 8603.792535] Lustre: server umount lustre-OST0001 complete [ 8612.300333] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8622.392919] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8666.243728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.815490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8666.892727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8667.034539] Lustre: lustre-MDT0000: new disk, initializing [ 8667.158529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.189246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8673.165963] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8684.402545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.623647] Lustre: 202865:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8684.692856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8684.697503] Lustre: Skipped 1 previous similar message [ 8684.776313] Lustre: lustre-MDT0001: new disk, initializing [ 8684.901238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8684.919479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8689.590871] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8697.573333] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8701.494961] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8708.881901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8709.219965] Lustre: 204607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8710.896172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8711.059675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8717.023779] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8725.895851] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8731.187817] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8731.496807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8737.123787] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8737.443440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8741.856843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8741.873172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.881790] LustreError: Skipped 1 previous similar message [ 8741.886744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8741.886753] Lustre: Skipped 8 previous similar messages [ 8741.922055] Lustre: Skipped 6 previous similar messages [ 8746.665821] Lustre: server umount lustre-OST0000 complete [ 8757.669357] Lustre: server umount lustre-MDT0000 complete [ 8761.313963] LustreError: 202876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.334706] LustreError: 202876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8761.667724] LustreError: 203642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229147 with bad export cookie 11156787282826424428 [ 8761.681702] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.686592] LustreError: 203642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8762.077317] Lustre: server umount lustre-MDT0001 complete [ 8766.755411] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:52:31 (1781229151) [ 8786.522887] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8799.597936] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8839.738695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.012572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8840.057638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8840.064900] Lustre: Skipped 1 previous similar message [ 8840.171124] Lustre: lustre-MDT0000: new disk, initializing [ 8840.173230] Lustre: Skipped 1 previous similar message [ 8840.290609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8840.295761] Lustre: Skipped 1 previous similar message [ 8845.858755] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8858.398346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.494501] Lustre: 209043:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8858.869055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8864.293287] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8872.978250] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8878.127428] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8887.440635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.874960] Lustre: 210785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8889.656200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8894.772823] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8903.696085] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8909.477976] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8909.838691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8915.245838] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8915.593764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8920.560479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.570149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8920.589018] LustreError: Skipped 2 previous similar messages [ 8920.592085] Lustre: Skipped 3 previous similar messages [ 8920.596943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8920.596953] Lustre: Skipped 6 previous similar messages [ 8932.319593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8932.756234] Lustre: server umount lustre-OST0000 complete [ 8943.404985] Lustre: server umount lustre-MDT0000 complete [ 8946.144670] LustreError: 209053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.176422] LustreError: 209053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8947.203621] LustreError: 209819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229333 with bad export cookie 11156787282826425310 [ 8947.205371] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.213297] LustreError: 209819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8947.637359] Lustre: server umount lustre-MDT0001 complete [ 8956.330192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.916413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.925966] Lustre: Skipped 5 previous similar messages [ 8961.598875] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8970.743580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.860430] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8981.769421] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8987.877270] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8997.549640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9005.975679] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9012.566722] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9026.016987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9026.029123] LustreError: Skipped 1 previous similar message [ 9026.034878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9026.050863] Lustre: Skipped 8 previous similar messages [ 9037.281374] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9037.529639] Lustre: server umount lustre-OST0000 complete [ 9047.904978] Lustre: server umount lustre-MDT0000 complete [ 9052.171156] LustreError: 212838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229438 with bad export cookie 11156787282826426185 [ 9052.177777] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9052.187799] LustreError: 212838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9052.528634] Lustre: server umount lustre-MDT0001 complete [ 9151.555788] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 9154.761485] Key type lgssc unregistered [ 9155.079846] LNet: 216163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9155.085601] LNetError: 216163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9155.096919] LNet: Removed LNI 192.168.202.148@tcp [ 9156.035160] Key type .llcrypt unregistered [ 9156.039062] Key type ._llcrypt unregistered [ 9180.341516] Key type ._llcrypt registered [ 9180.342941] Key type .llcrypt registered [ 9180.498115] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 9197.749623] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9198.972262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9199.234645] alg: No test for adler32 (adler32-zlib) [ 9200.340829] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [ 9200.607973] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 9202.375683] Key type lgssc registered [ 9203.661402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9241.994986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9242.015046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.337982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9243.361961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9243.475502] Lustre: lustre-MDT0000: new disk, initializing [ 9243.567373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9243.592496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9248.625362] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9261.026899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.201447] Lustre: 219486:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9261.310949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9261.315134] Lustre: Skipped 1 previous similar message [ 9261.429588] Lustre: lustre-MDT0001: new disk, initializing [ 9261.554854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9261.660476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9261.687527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9266.320496] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9275.311957] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9280.081838] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9288.121859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9288.523870] Lustre: lustre-OST0000: new disk, initializing [ 9288.533681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9288.554889] Lustre: 221237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9288.735235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9294.921760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9294.935394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9295.090548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9296.149460] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9304.371314] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9308.690561] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9308.994676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9314.105979] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9314.340383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9320.930406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9320.937691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.950542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9320.953635] Lustre: Skipped 1 previous similar message [ 9320.966751] Lustre: Skipped 1 previous similar message [ 9322.672059] Lustre: server umount lustre-OST0000 complete [ 9328.096818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9328.111876] LustreError: Skipped 1 previous similar message [ 9328.119562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.142778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.168784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.171373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.212038] Lustre: Skipped 1 previous similar message [ 9336.288301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.306790] Lustre: Skipped 1 previous similar message [ 9340.401771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.412681] Lustre: Skipped 1 previous similar message [ 9341.407225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9341.916161] Lustre: server umount lustre-MDT0000 complete [ 9346.569131] LustreError: 219476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229732 with bad export cookie 4742521489374934382 [ 9346.581946] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9346.582746] LustreError: 219476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9346.951753] Lustre: server umount lustre-MDT0001 complete [ 9367.263956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9378.809374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.445422] LustreError: 224008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.462342] LustreError: 224008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9379.495259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.364329] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9384.944984] LustreError: 224009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9390.050994] LustreError: 224008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.884876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.349509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9399.058625] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9402.494731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9410.129457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9410.583495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.640193] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9427.805350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9427.928866] Lustre: lustre-OST0001: new disk, initializing [ 9427.931241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9427.936332] Lustre: 226265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9428.035878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9435.223738] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9437.331803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9437.341630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9437.535279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9448.499703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9456.125640] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9469.869508] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:04:14 (1781229854) [ 9473.514541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9473.541571] Lustre: Skipped 2 previous similar messages [ 9473.542491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9473.557717] Lustre: Skipped 3 previous similar messages [ 9478.396845] Lustre: server umount lustre-MDT0000 complete [ 9478.629954] LustreError: 224009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.663645] LustreError: 224009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9482.263438] LustreError: 223988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229868 with bad export cookie 4742521489374935264 [ 9482.274367] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9482.275392] LustreError: 223988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9482.696576] Lustre: server umount lustre-MDT0001 complete [ 9497.284786] Lustre: server umount lustre-OST0000 complete [ 9499.105377] Lustre: 217710:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229869/real 1781229869] req@ffff9770983a7480 x1867754594483712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781229885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9499.146262] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.167680] Lustre: Skipped 1 previous similar message [ 9502.172331] Lustre: server umount lustre-OST0001 complete [ 9510.086808] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 9522.083706] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9566.767130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9566.963228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9566.989098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9567.045888] Lustre: lustre-MDT0000: new disk, initializing [ 9567.125441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9567.141470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9571.831156] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9583.617942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.822916] Lustre: 230162:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9583.912831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9583.919033] Lustre: Skipped 1 previous similar message [ 9584.045124] Lustre: lustre-MDT0001: new disk, initializing [ 9584.286785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9584.305441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9589.629712] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9599.205564] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9603.763602] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9611.085773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9611.470075] Lustre: lustre-OST0000: new disk, initializing [ 9611.477144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9611.490520] Lustre: 231907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9613.246672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9613.269263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9613.373291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9617.823188] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9626.596624] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9632.433898] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9632.642639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9637.378901] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9637.630947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9644.002430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.004314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9644.034757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9644.046585] Lustre: Skipped 1 previous similar message [ 9654.246369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9654.520620] Lustre: server umount lustre-OST0000 complete [ 9664.478230] Lustre: server umount lustre-MDT0000 complete [ 9664.492071] LustreError: 230171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9664.511295] LustreError: 230171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9668.303751] LustreError: 230154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230054 with bad export cookie 4742521489374937392 [ 9668.310477] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9668.324536] LustreError: 230154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9668.659851] Lustre: server umount lustre-MDT0001 complete [ 9677.686953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.071463] LustreError: 233976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.162395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.170643] Lustre: Skipped 2 previous similar messages [ 9683.452404] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9693.663235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.954302] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9708.666202] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9713.713347] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9722.432328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.535662] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9736.297327] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9887.712819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9887.719843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.721021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.730989] Lustre: Skipped 4 previous similar messages [ 9887.740943] Lustre: Skipped 7 previous similar messages [ 9889.512967] Lustre: server umount lustre-OST0000 complete [ 9908.191151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9908.481383] Lustre: server umount lustre-MDT0000 complete [ 9912.297224] LustreError: 234687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.331813] LustreError: 234687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9913.012174] LustreError: 233958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230298 with bad export cookie 4742521489374938281 [ 9913.014277] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.024971] LustreError: 233958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9913.532340] Lustre: server umount lustre-MDT0001 complete [10003.449427] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [10006.753194] Key type lgssc unregistered [10007.019254] LNet: 237229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10007.029548] LNetError: 237229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10007.057932] LNet: Removed LNI 192.168.202.148@tcp [10007.977159] Key type .llcrypt unregistered [10007.979823] Key type ._llcrypt unregistered [10034.314859] Key type ._llcrypt registered [10034.317534] Key type .llcrypt registered [10034.467966] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [10050.166879] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10051.686251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10051.942853] alg: No test for adler32 (adler32-zlib) [10053.055504] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [10053.331978] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [10055.079181] Key type lgssc registered [10056.796377] Lustre: Echo OBD driver; http://www.lustre.org/ [10100.389805] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10108.875742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10108.919817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.269370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10110.308837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10110.450272] Lustre: lustre-MDT0000: new disk, initializing [10110.575672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.594928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10115.204982] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10126.847479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.024578] Lustre: 241247:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10127.112777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10127.116717] Lustre: Skipped 1 previous similar message [10127.231623] Lustre: lustre-MDT0001: new disk, initializing [10127.378190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.422794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10127.437307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10132.057641] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10137.310648] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10143.468037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10144.045343] Lustre: lustre-OST0000: new disk, initializing [10144.052947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10144.074805] Lustre: 242843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10144.286120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10146.504589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10146.523983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10146.620572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10152.401906] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10164.461825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.652107] Lustre: lustre-OST0001: new disk, initializing [10164.656901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10164.664357] Lustre: 243698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10164.770689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10169.991481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10170.016069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10170.142796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10170.882849] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10182.860185] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10191.057795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10204.128471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.139559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.172670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10206.186114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10206.191171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.197633] Lustre: Skipped 1 previous similar message [10206.226195] Lustre: Skipped 1 previous similar message [10209.040779] Lustre: server umount lustre-MDT0000 complete [10212.382073] LustreError: 241239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230598 with bad export cookie 4208932020271306103 [10212.395672] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10212.690912] Lustre: server umount lustre-MDT0001 complete [10226.813365] Lustre: server umount lustre-OST0000 complete [10241.337687] Lustre: server umount lustre-OST0001 complete [10254.806208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10263.658163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10271.161824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10281.520475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10303.668255] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10315.481986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.507838] Lustre: lustre-MDT0000: reset Object Index mappings [10316.071042] LustreError: 248677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.082732] LustreError: 248677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10316.154091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10321.363715] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10321.379816] LustreError: 248678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.478362] LustreError: 248677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.271762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.289269] Lustre: lustre-MDT0001: reset Object Index mappings [10330.599896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10330.610180] Lustre: Skipped 2 previous similar messages [10330.820573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10335.218996] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10338.410651] Lustre: 249785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10344.139486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10349.863701] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10355.762080] LustreError: 250141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.932023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10359.192642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10359.198485] Lustre: Skipped 1 previous similar message [10360.275706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10360.298430] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10365.383253] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10374.097637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10378.546353] Lustre: 251623:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10385.763398] Lustre: server umount lustre-MDT0000 complete [10385.894142] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.903296] Lustre: Skipped 4 previous similar messages [10385.907319] LustreError: 248672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.967939] LustreError: 248672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10390.172776] LustreError: 248659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230776 with bad export cookie 4208932020271312893 [10390.174471] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10390.181094] LustreError: 248659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10390.518336] Lustre: server umount lustre-MDT0001 complete [10404.480891] Lustre: server umount lustre-OST0000 complete [10406.371137] Lustre: 238771:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230776/real 1781230776] req@ffff976f8fddf840 x1867755488933760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781230792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10406.412960] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.805376] Lustre: server umount lustre-OST0001 complete [10428.136317] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:20:12 (1781230812) [10443.348467] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [10453.768634] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10489.969954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.120763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10490.139708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10490.184741] Lustre: lustre-MDT0000: new disk, initializing [10490.279559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.307059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10495.412503] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10505.142241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.230250] Lustre: 255840:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10505.280655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10505.283467] Lustre: Skipped 1 previous similar message [10505.380021] Lustre: lustre-MDT0001: new disk, initializing [10505.530080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10505.551311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10509.844378] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10517.148525] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10520.922261] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10527.361546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.591843] Lustre: lustre-OST0000: new disk, initializing [10527.597413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10527.608549] Lustre: 257583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10529.613562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10529.626700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10529.766640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10533.423797] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10540.201891] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10543.964733] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10544.128556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10547.997533] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10548.210222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10550.243289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10550.253530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.265104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10550.284127] Lustre: Skipped 1 previous similar message [10555.385191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10555.398987] Lustre: Skipped 1 previous similar message [10556.615958] Lustre: server umount lustre-OST0000 complete [10562.016401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10562.029647] LustreError: Skipped 1 previous similar message [10562.035262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.603462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10565.611891] Lustre: Skipped 2 previous similar messages [10567.880667] Lustre: server umount lustre-MDT0000 complete [10570.729824] LustreError: 258480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.752407] LustreError: 258480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10571.876658] LustreError: 255834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230957 with bad export cookie 4208932020271314986 [10571.877667] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10571.880059] LustreError: 255834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10572.153957] Lustre: server umount lustre-MDT0001 complete [10589.616419] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10600.250191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.660780] LustreError: 260355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.759735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.764332] Lustre: Skipped 2 previous similar messages [10605.616525] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10613.500141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.922193] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10620.958197] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10628.754972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10636.294152] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10646.453984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10646.667665] Lustre: lustre-OST0001: new disk, initializing [10646.678761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10646.685843] Lustre: 262612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10647.885235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10647.903951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10648.020699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10652.918274] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10662.121699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10665.630193] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10670.583461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10670.696921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10676.239571] Lustre: server umount lustre-MDT0000 complete [10678.756513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.766151] Lustre: Skipped 3 previous similar messages [10678.783978] LustreError: 260355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.812065] LustreError: 260355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10679.689543] LustreError: 260334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231065 with bad export cookie 4208932020271315868 [10679.698518] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10679.702629] LustreError: 260334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10682.032994] Lustre: server umount lustre-MDT0001 complete [10695.584381] Lustre: server umount lustre-OST0000 complete [10700.129985] Lustre: 238773:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231069/real 1781231069] req@ffff9770bde3f480 x1867755489031552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10700.916051] Lustre: server umount lustre-OST0001 complete [10719.417370] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10729.480397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.093809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10730.106754] Lustre: Skipped 3 previous similar messages [10735.125064] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10744.071062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.288745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10749.245560] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10752.565565] Lustre: 266737:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10758.989736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.053524] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10769.711714] LustreError: 267090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.747465] LustreError: 267090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10773.914778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.242936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10775.339554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10781.456174] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10789.712436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10793.116332] Lustre: 268580:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10797.970107] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10798.077358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10806.738640] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:26:31 (1781231191) [10826.219926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10826.248108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.258106] Lustre: Skipped 2 previous similar messages [10826.264596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10829.078433] Lustre: server umount lustre-MDT0000 complete [10832.700414] LustreError: 267901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231218 with bad export cookie 4208932020271322210 [10832.708558] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10832.714410] LustreError: 267901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10839.261602] Lustre: server umount lustre-MDT0001 complete [10858.975329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10859.279759] Lustre: server umount lustre-OST0000 complete [10870.376451] Lustre: server umount lustre-OST0001 complete [10877.695492] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [10888.173494] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10933.469421] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10944.336041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.657730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10944.698774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10944.781468] Lustre: lustre-MDT0000: new disk, initializing [10944.896545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10948.779578] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10960.058139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10960.286620] Lustre: 272774:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10960.692688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10965.797066] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10978.865961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.204903] Lustre: 274370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10981.386738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10986.027890] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10998.285288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10998.602926] Lustre: 275224:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10998.767577] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10998.771307] Lustre: Skipped 6 previous similar messages [11000.786513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11000.805564] Lustre: Skipped 1 previous similar message [11000.975861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11005.726235] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11015.677814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11019.466649] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11019.471778] Lustre: Skipped 1 previous similar message [11033.864491] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:30:18 (1781231418) [11036.642731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.651548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11036.669218] Lustre: Skipped 6 previous similar messages [11036.675619] Lustre: Skipped 7 previous similar messages [11042.334278] Lustre: server umount lustre-MDT0000 complete [11046.047487] LustreError: 272766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231431 with bad export cookie 4208932020271330099 [11046.064484] LustreError: 272766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11046.068650] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11046.638379] Lustre: server umount lustre-MDT0001 complete [11060.767351] Lustre: server umount lustre-OST0000 complete [11062.242079] Lustre: 238773:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231432/real 1781231432] req@ffff9770898fe940 x1867755489180160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11062.282372] Lustre: 238773:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11065.148871] Lustre: server umount lustre-OST0001 complete [11074.070216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11074.495355] LustreError: 277456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.512433] LustreError: 277456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [11078.992341] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11087.180338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.928790] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11100.142672] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11104.318352] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11111.778350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11120.472936] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11126.883217] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11144.720191] loop1: detected capacity change from 0 to 204800000 [11144.977929] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11144.984907] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11144.995810] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11150.782625] loop1: detected capacity change from 0 to 204800000 [11157.852366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.018488] LustreError: 277444:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs29906: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11158.031770] LustreError: 277444:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs29906 [11158.041647] LustreError: 277444:0:(mgs_handler.c:517:mgs_target_reg()) fs29906-MDT0001: Error checking target index: rc = -13 [11158.076984] LustreError: fs29906-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11158.097098] LustreError: 280836:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11158.105560] LustreError: 280836:0:(tgt_mount.c:2084:server_put_super()) no obd fs29906-MDT0001 [11158.130538] LustreError: 280836:0:(tgt_mount.c:116:server_deregister_mount()) fs29906-MDT0001 not registered [11158.187914] Lustre: server umount fs29906-MDT0001 complete [11158.195515] LustreError: 280836:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11167.556796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.661463] Lustre: Setting parameter fs29906-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29906-MDT0001 [11167.724886] Lustre: srv-fs29906-MDT0001: No data found on store. Initialize space. [11167.728526] Lustre: Skipped 4 previous similar messages [11167.829167] Lustre: fs29906-MDT0001: new disk, initializing [11167.831251] Lustre: Skipped 3 previous similar messages [11167.931182] Lustre: srv-fs29906-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11167.935040] Lustre: 281305:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29906-MDT0001-mdtlov: can't create new llog: rc = -115 [11167.939865] LustreError: 281305:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29906-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11172.595918] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11180.332941] Lustre: server umount fs29906-MDT0001 complete [11193.848590] loop1: detected capacity change from 0 to 204800000 [11197.413068] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11200.012968] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11200.444791] Lustre: 282853:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29906-MDT0001-mdtlov: can't create new llog: rc = -115 [11200.456409] LustreError: 282853:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29906-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11204.927189] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11209.591425] Lustre: server umount fs29906-MDT0001 complete [11219.953606] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [11219.961128] Lustre: Skipped 7 previous similar messages [11220.966344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11220.978811] LustreError: Skipped 1 previous similar message [11220.983798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.001082] Lustre: Skipped 3 previous similar messages [11224.745493] Lustre: server umount lustre-MDT0000 complete [11233.994076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.064689] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11238.497148] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11239.421522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11248.011900] loop1: detected capacity change from 0 to 204800000 [11253.303435] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.833852] Lustre: 285266:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29906-MDT0001-mdtlov: can't create new llog: rc = -115 [11253.839284] LustreError: 285266:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29906-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11258.233167] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11263.458283] Lustre: server umount fs29906-MDT0001 complete [11287.520838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11293.809901] Lustre: server umount lustre-OST0000 complete [11305.281382] Lustre: server umount lustre-MDT0000 complete [11309.953805] LustreError: 277438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231695 with bad export cookie 4208932020271332780 [11309.955934] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11309.961315] LustreError: 277438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11310.425224] Lustre: server umount lustre-MDT0001 complete [11322.527904] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [11326.148480] Key type lgssc unregistered [11326.500932] LNet: 287576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11326.511689] LNetError: 287576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11327.589715] LNet: Removed LNI 192.168.202.148@tcp [11328.319156] Key type .llcrypt unregistered [11328.320612] Key type ._llcrypt unregistered [11348.057988] Key type ._llcrypt registered [11348.060481] Key type .llcrypt registered [11348.182203] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11349.772431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.812052] alg: No test for adler32 (adler32-zlib) [11350.868441] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [11351.177056] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [11352.865045] Key type lgssc registered [11354.039909] Lustre: Echo OBD driver; http://www.lustre.org/ [11361.706354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11361.723864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.254914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11368.910317] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11373.548093] LustreError: 288833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.582205] LustreError: 288833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11377.934766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.307303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11383.599822] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11389.563663] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11394.740419] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11402.280583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11402.803755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11403.816717] LustreError: 290435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.870569] LustreError: 290435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.519600] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11415.017659] LustreError: 290805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.043586] LustreError: 290805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11416.788672] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11419.508352] LustreError: 290435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.874906] LustreError: 290497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.898606] LustreError: 290497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11437.511143] loop1: detected capacity change from 0 to 204800000 [11437.639822] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11437.650622] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11437.659520] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11443.527526] loop1: detected capacity change from 0 to 204800000 [11449.844167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.073617] Lustre: fs22220-OST0000: new disk, initializing [11450.077919] Lustre: srv-fs22220-OST0000: No data found on store. Initialize space. [11450.085322] Lustre: 292229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11450.188606] Lustre: fs22220-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11450.324961] LustreError: 290437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.365128] LustreError: 290437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11455.635668] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11465.540078] Lustre: server umount fs22220-OST0000 complete [11482.671970] loop1: detected capacity change from 0 to 204800000 [11486.169436] LustreError: 290435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11486.207249] LustreError: 290435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [11486.717240] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11489.089589] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.441560] Lustre: fs22220-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11491.816176] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11496.326525] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11505.585395] loop2: detected capacity change from 0 to 204800000 [11505.975761] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11505.995563] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11506.028435] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11511.134148] loop2: detected capacity change from 0 to 204800000 [11517.195898] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11517.294630] LustreError: fs22220-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11517.303319] LustreError: fs22220-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11517.309268] LustreError: 295236:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11517.314783] LustreError: 295236:0:(tgt_mount.c:2084:server_put_super()) no obd fs22220-OST0001 [11517.318883] LustreError: 295236:0:(tgt_mount.c:116:server_deregister_mount()) fs22220-OST0001 not registered [11517.345316] Lustre: server umount fs22220-OST0001 complete [11517.346934] LustreError: 295236:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11528.509165] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11528.595341] Lustre: fs22220-OST0001: new disk, initializing [11528.597909] Lustre: srv-fs22220-OST0001: No data found on store. Initialize space. [11528.603795] Lustre: 295787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11528.653806] Lustre: fs22220-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11538.555444] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11551.075671] Lustre: server umount fs22220-OST0000 complete [11552.742495] LustreError: 290805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.757551] LustreError: 290805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11563.236463] Lustre: server umount fs22220-OST0001 complete [11577.639318] loop1: detected capacity change from 0 to 204800000 [11584.665697] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11584.995673] Lustre: fs22220-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.111788] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11600.201159] loop2: detected capacity change from 0 to 204800000 [11607.131574] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11615.215665] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11624.681509] Lustre: server umount fs22220-OST0000 complete [11634.358375] Lustre: server umount fs22220-OST0001 complete [11640.306934] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11645.415859] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11649.504880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11649.510325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.532344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11650.532109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11650.534938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.898457] Lustre: server umount lustre-OST0000 complete [11659.744476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11659.752265] LustreError: Skipped 1 previous similar message [11659.757278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.772291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11659.779255] Lustre: Skipped 1 previous similar message [11663.122927] Lustre: server umount lustre-MDT0000 complete [11666.831737] LustreError: 288814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232052 with bad export cookie 11262851496844446643 [11666.852296] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.263547] Lustre: server umount lustre-MDT0001 complete [11681.102695] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [11684.764108] Key type lgssc unregistered [11685.166780] LNet: 301646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11685.174748] LNetError: 301646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11685.209241] LNet: Removed LNI 192.168.202.148@tcp [11686.206345] Key type .llcrypt unregistered [11686.208674] Key type ._llcrypt unregistered [11725.238713] Key type ._llcrypt registered [11725.241946] Key type .llcrypt registered [11725.354567] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [11739.974487] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11741.450263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11741.496087] alg: No test for adler32 (adler32-zlib) [11742.547755] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [11742.827971] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [11744.575195] Key type lgssc registered [11745.806940] Lustre: Echo OBD driver; http://www.lustre.org/ [11781.274403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11781.297778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11782.509201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11782.536640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11782.646857] Lustre: lustre-MDT0000: new disk, initializing [11782.737958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11782.778378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11787.617533] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11798.357831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.471092] Lustre: 305473:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11798.524893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11798.529408] Lustre: Skipped 1 previous similar message [11798.630291] Lustre: lustre-MDT0001: new disk, initializing [11798.768593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11798.820100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11798.831388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11803.559166] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11811.659663] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11816.067922] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11824.742573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11825.178622] Lustre: lustre-OST0000: new disk, initializing [11825.188670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11825.199898] Lustre: 307213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11825.337659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11832.337696] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11834.436195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11834.452376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11834.585053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11841.621630] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11844.716214] Lustre: Mounted lustre-client [11845.951279] Lustre: Unmounted lustre-client [11859.942863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.944283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11859.950693] Lustre: Skipped 1 previous similar message [11859.966531] Lustre: Skipped 1 previous similar message [11862.749508] Lustre: server umount lustre-OST0000 complete [11870.185996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11870.188545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11870.208885] Lustre: Skipped 1 previous similar message [11870.223570] Lustre: Skipped 1 previous similar message [11873.647490] Lustre: server umount lustre-MDT0000 complete [11878.732877] LustreError: 307212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232264 with bad export cookie 11268780358946874924 [11878.738491] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11878.745378] LustreError: 307212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11879.286696] Lustre: server umount lustre-MDT0001 complete [11891.576916] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [11895.105951] Key type lgssc unregistered [11898.529816] LNet: 309245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11898.537898] LNetError: 309245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11974.314830] LNet: Removed LNI 192.168.202.148@tcp [11975.174901] Key type .llcrypt unregistered [11975.180554] Key type ._llcrypt unregistered [11991.685175] Key type ._llcrypt registered [11991.686467] Key type .llcrypt registered [11991.825126] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [12007.871255] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12009.682739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12009.791262] alg: No test for adler32 (adler32-zlib) [12010.957892] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [12011.270973] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [12012.951356] Key type lgssc registered [12014.489780] Lustre: Echo OBD driver; http://www.lustre.org/ [12051.199172] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:47:15 (1781232435) [12059.075542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12059.131106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12060.442272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12060.478243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12060.596509] Lustre: lustre-MDT0000: new disk, initializing [12060.735904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12060.780189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12066.404375] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12077.910390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.992778] Lustre: 312430:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12078.033138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12078.035162] Lustre: Skipped 1 previous similar message [12078.076134] Lustre: lustre-MDT0001: new disk, initializing [12078.129085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12078.204735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12078.214783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12083.693158] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12091.759338] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12096.647351] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12104.466873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12104.758470] Lustre: lustre-OST0000: new disk, initializing [12104.761963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12104.770861] Lustre: 314176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12104.879389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12111.617871] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12112.006793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12112.041086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12112.173876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12119.856965] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12132.832927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12132.853117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.857690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.876765] Lustre: Skipped 1 previous similar message [12135.165569] Lustre: server umount lustre-OST0000 complete [12139.489650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12139.492163] LustreError: Skipped 1 previous similar message [12139.495336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.508767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12139.519116] Lustre: Skipped 1 previous similar message [12143.078221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.079355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12143.104528] Lustre: Skipped 1 previous similar message [12145.795784] Lustre: server umount lustre-MDT0000 complete [12149.220170] LustreError: 312425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232535 with bad export cookie 2719588507110664660 [12149.222342] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12149.227459] LustreError: 312425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12149.563892] Lustre: server umount lustre-MDT0001 complete [12163.484251] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12166.567664] Key type lgssc unregistered [12167.022648] LNet: 315807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12167.034705] LNetError: 315807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12168.105828] LNet: Removed LNI 192.168.202.148@tcp [12169.105082] Key type .llcrypt unregistered [12169.107323] Key type ._llcrypt unregistered [12196.066047] Key type ._llcrypt registered [12196.069803] Key type .llcrypt registered [12196.182528] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12197.411699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12197.534815] alg: No test for adler32 (adler32-zlib) [12198.737444] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [12199.110174] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [12200.783200] Key type lgssc registered [12201.993070] Lustre: Echo OBD driver; http://www.lustre.org/ [12209.671947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12209.690668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12211.250567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12216.223183] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12221.415662] LustreError: 317213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.432230] LustreError: 317213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12225.491345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.781221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12229.413698] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12234.441805] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12237.890725] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12243.862581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12244.235035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12249.587901] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12254.961782] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12266.634107] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12268.401301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12276.216399] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [12277.217374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12277.217675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12277.238821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.662577] Lustre: server umount lustre-MDT0000 complete [12281.333753] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.351578] LustreError: 317208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.398383] LustreError: 317208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12283.407908] LustreError: 317195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232669 with bad export cookie 7973653150516520640 [12283.412863] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12283.415239] LustreError: 317195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12286.425839] LustreError: 317210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12286.457144] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [12286.476052] Lustre: Skipped 2 previous similar messages [12287.471395] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.498106] Lustre: Skipped 2 previous similar messages [12288.483966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12288.487623] Lustre: Skipped 1 previous similar message [12289.892557] Lustre: server umount lustre-MDT0001 complete [12293.475706] Lustre: server umount lustre-OST0000 complete [12303.786200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.758129] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12314.736360] Lustre: 320940:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.148@tcp [12317.018256] Lustre: server umount MGS complete [12324.775400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12325.182208] LustreError: 321512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.205326] LustreError: 321512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12325.321873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12330.441771] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12335.590862] LustreError: 321513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.598503] LustreError: 321513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12338.396455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.738985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12343.244211] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12351.289631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.586040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12357.536513] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12363.912255] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12365.931702] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12370.367499] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12373.321690] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [12379.105757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12379.122742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12379.134814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12381.157559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12381.168728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.548797] Lustre: server umount lustre-OST0000 complete [12389.859467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12389.865992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.888468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12389.891824] Lustre: Skipped 2 previous similar messages [12394.829250] Lustre: server umount lustre-MDT0000 complete [12396.513620] LustreError: 321509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12396.551636] LustreError: 321509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12398.746867] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232784 with bad export cookie 7973653150516521480 [12398.762193] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12398.763408] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.085885] Lustre: server umount lustre-MDT0001 complete [12409.960588] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12413.025381] Key type lgssc unregistered [12413.416239] LNet: 324687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12413.421953] LNetError: 324687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12413.436468] LNet: Removed LNI 192.168.202.148@tcp [12414.417153] Key type .llcrypt unregistered [12414.419680] Key type ._llcrypt unregistered [12434.035605] Key type ._llcrypt registered [12434.037359] Key type .llcrypt registered [12434.163779] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [12450.836801] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12452.081199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12452.130453] alg: No test for adler32 (adler32-zlib) [12453.561474] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [12453.968649] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [12455.735144] Key type lgssc registered [12456.893565] Lustre: Echo OBD driver; http://www.lustre.org/ [12492.107906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12492.135449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.558543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12493.609083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12493.716345] Lustre: lustre-MDT0000: new disk, initializing [12493.889122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12493.939282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12498.414980] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12509.038087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.168299] Lustre: 327815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12509.216803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12509.223829] Lustre: Skipped 1 previous similar message [12509.281160] Lustre: lustre-MDT0001: new disk, initializing [12509.360498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.413534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12509.424597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12513.845464] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12521.340447] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12524.848544] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12531.835150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.087277] Lustre: lustre-OST0000: new disk, initializing [12532.089509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12532.092777] Lustre: 329556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12532.142610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.363522] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12541.072056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12541.095069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12541.250161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12546.040814] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12551.121918] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12551.389927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12556.920541] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12557.135726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12561.892613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12561.899439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.903182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.910054] Lustre: Skipped 1 previous similar message [12567.010694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12567.023543] Lustre: Skipped 2 previous similar messages [12572.135300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12572.143882] Lustre: Skipped 1 previous similar message [12573.151468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12573.327894] Lustre: server umount lustre-OST0000 complete [12581.347141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12581.352639] LustreError: Skipped 1 previous similar message [12581.354185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12581.366328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12582.371090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12585.987748] Lustre: server umount lustre-MDT0000 complete [12590.302443] LustreError: 327807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232976 with bad export cookie 11282011861197658707 [12590.340934] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12590.676373] Lustre: server umount lustre-MDT0001 complete [12602.752340] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:56:27 (1781232987) [12619.691497] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [12629.211371] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12667.486401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.729657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12667.755454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12667.834422] Lustre: lustre-MDT0000: new disk, initializing [12667.944391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12667.962090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12672.384948] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12684.635295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.785376] Lustre: 334138:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12684.858446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12684.867903] Lustre: Skipped 1 previous similar message [12684.978729] Lustre: lustre-MDT0001: new disk, initializing [12685.109157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12685.139273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12685.148348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12689.878375] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12697.842986] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12701.598910] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12708.248860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12708.471901] Lustre: lustre-OST0000: new disk, initializing [12708.479755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12708.485288] Lustre: 335879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12708.565829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12713.810966] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12717.645693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12717.657861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12717.755116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12720.328577] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12759.012562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.024155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12759.040144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12759.047147] Lustre: Skipped 3 previous similar messages [12762.392425] Lustre: server umount lustre-OST0000 complete [12767.200267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12767.207603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.221677] Lustre: Skipped 1 previous similar message [12767.242265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12773.075898] Lustre: server umount lustre-MDT0000 complete [12774.372519] LustreError: 336633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12774.387576] LustreError: 336633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12777.089054] LustreError: 334130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233162 with bad export cookie 11282011861197659589 [12777.105787] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12777.106341] LustreError: 334130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12777.546261] Lustre: server umount lustre-MDT0001 complete [12788.117931] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12790.289835] Key type lgssc unregistered [12790.528167] LNet: 337925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12790.536810] LNetError: 337925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12790.556503] LNet: Removed LNI 192.168.202.148@tcp [12791.214275] Key type .llcrypt unregistered [12791.219515] Key type ._llcrypt unregistered [12886.157154] Key type ._llcrypt registered [12886.160509] Key type .llcrypt registered [12886.321557] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12886.612346] Key type .llcrypt unregistered [12886.615649] Key type ._llcrypt unregistered [12972.078731] Key type ._llcrypt registered [12972.080799] Key type .llcrypt registered [12972.176920] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12972.429582] Key type .llcrypt unregistered [12972.432117] Key type ._llcrypt unregistered [13006.041394] Key type ._llcrypt registered [13006.046150] Key type .llcrypt registered [13006.106989] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13006.271356] Key type .llcrypt unregistered [13006.273078] Key type ._llcrypt unregistered [13041.181927] Key type ._llcrypt registered [13041.188768] Key type .llcrypt registered [13041.372438] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13041.723583] Key type .llcrypt unregistered [13041.729049] Key type ._llcrypt unregistered [13127.203404] Key type ._llcrypt registered [13127.207190] Key type .llcrypt registered [13127.340271] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13127.622311] Key type .llcrypt unregistered [13127.623950] Key type ._llcrypt unregistered [13162.075432] Key type ._llcrypt registered [13162.077676] Key type .llcrypt registered [13162.180626] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13162.503409] Key type .llcrypt unregistered [13162.505058] Key type ._llcrypt unregistered [13194.491897] Key type ._llcrypt registered [13194.493993] Key type .llcrypt registered [13194.584988] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13194.982960] Key type .llcrypt unregistered [13194.986161] Key type ._llcrypt unregistered [13224.192109] Key type ._llcrypt registered [13224.193622] Key type .llcrypt registered [13224.361264] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13224.612470] Key type .llcrypt unregistered [13224.616533] Key type ._llcrypt unregistered [13256.601107] Key type ._llcrypt registered [13256.603567] Key type .llcrypt registered [13256.687764] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13256.934482] Key type .llcrypt unregistered [13256.937428] Key type ._llcrypt unregistered [13288.111291] Key type ._llcrypt registered [13288.113113] Key type .llcrypt registered [13288.196392] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13288.564402] Key type .llcrypt unregistered [13288.566066] Key type ._llcrypt unregistered [13349.519461] Key type ._llcrypt registered [13349.521912] Key type .llcrypt registered [13349.643059] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13350.019426] Key type .llcrypt unregistered [13350.031388] Key type ._llcrypt unregistered [13384.275912] Key type ._llcrypt registered [13384.278282] Key type .llcrypt registered [13384.393205] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13384.762370] Key type .llcrypt unregistered [13384.765196] Key type ._llcrypt unregistered [13418.156257] Key type ._llcrypt registered [13418.157535] Key type .llcrypt registered [13418.353830] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13418.768381] Key type .llcrypt unregistered [13418.771693] Key type ._llcrypt unregistered [13473.737668] Key type ._llcrypt registered [13473.741826] Key type .llcrypt registered [13473.838982] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [13492.771612] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [13494.490230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13494.573304] alg: No test for adler32 (adler32-zlib) [13495.596238] Lustre: Lustre: Build Version: 2.17.53_71_gce50822 [13495.894503] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [13497.679596] Key type lgssc registered [13499.268620] Lustre: Echo OBD driver; http://www.lustre.org/ [13537.154523] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13537.181109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13538.541392] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13538.571283] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13538.712091] Lustre: tst250fs-MDT0000: new disk, initializing [13538.867133] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13538.897303] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13543.715348] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13555.005814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13555.167791] Lustre: 358029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13555.231633] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13555.235497] Lustre: Skipped 1 previous similar message [13555.300198] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [13555.377379] Lustre: tst250fs-MDT0001: new disk, initializing [13555.507168] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13555.562693] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13555.579246] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13560.266272] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13565.636924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13573.779810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13574.131558] Lustre: tst250fs-OST0000: new disk, initializing [13574.136587] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13574.147277] Lustre: 359625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13574.279018] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13577.791813] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13577.801406] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13577.908603] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13581.587634] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13593.612917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13593.947685] Lustre: tst250fs-OST0001: new disk, initializing [13593.953653] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13593.965227] Lustre: 360479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13594.090921] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13600.525863] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13601.393975] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13601.433931] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13601.526506] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13607.126850] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13609.217358] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13609.226662] Lustre: Skipped 1 previous similar message [13632.484687] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13632.491367] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13632.501515] Lustre: Skipped 1 previous similar message [13632.501976] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13632.523370] Lustre: Skipped 3 previous similar messages [13637.609599] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13637.613664] Lustre: Skipped 2 previous similar messages [13642.730602] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13642.744476] Lustre: Skipped 4 previous similar messages [13644.255156] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13644.518487] Lustre: server umount tst250fs-MDT0000 complete [13648.648348] LustreError: 358021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234034 with bad export cookie 5788688692041939424 [13648.657405] LustreError: 358021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13648.658060] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13649.352867] Lustre: server umount tst250fs-MDT0001 complete [13664.285382] Lustre: server umount tst250fs-OST0000 complete [13668.320358] Lustre: 356252:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234038/real 1781234038] req@ffff976f8fed3480 x1867759098080128/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781234054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13668.357742] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13668.374742] Lustre: Skipped 2 previous similar messages [13669.045070] Lustre: server umount tst250fs-OST0001 complete [13693.099371] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [13703.105437] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [13736.179429] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:15:21 (1781234121) [13737.678925] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13756.509291] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [13769.270385] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [13799.970771] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13475 sec ========= 23:16:24 (1781234184) [13801.847401] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:16:26 (1781234186) === [13805.727964] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:16:30 (1781234190) === [13844.155735] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13847.113623] Key type lgssc unregistered [13847.331993] LNet: 369525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13847.370693] LNetError: 369525:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13847.412849] LNet: Removed LNI 192.168.202.148@tcp [13848.556300] Key type .llcrypt unregistered [13848.558446] Key type ._llcrypt unregistered