[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450316092 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002406] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009162] LSM: Security Framework initializing [ 0.010074] Yama: becoming mindful. [ 0.011051] SELinux: Initializing. [ 0.012095] *** VALIDATE selinux *** [ 0.021349] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026612] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030029] *** VALIDATE tmpfs *** [ 0.031432] *** VALIDATE proc *** [ 0.032271] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035225] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037044] Spectre V2 : User space: Vulnerable [ 0.038016] Speculative Store Bypass: Vulnerable [ 0.041838] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.043867] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044785] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045033] ... version: 2 [ 0.046037] ... bit width: 48 [ 0.047019] ... generic registers: 4 [ 0.048020] ... value mask: 0000ffffffffffff [ 0.049024] ... max period: 00007fffffffffff [ 0.050022] ... fixed-purpose events: 3 [ 0.051021] ... event mask: 000000070000000f [ 0.053308] rcu: Hierarchical SRCU implementation. [ 0.055723] smp: Bringing up secondary CPUs ... [ 0.056716] x86: Booting SMP configuration: [ 0.057040] .... node #0, CPUs: #1 #2 #3 [ 0.061165] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.064028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222075] node 0 deferred pages initialised in 156ms [ 0.225242] devtmpfs: initialized [ 0.226259] x86/mm: Memory block size: 128MB [ 0.228982] gcov: version magic: 0x41383552 [ 0.230445] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.234083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.237326] pinctrl core: initialized pinctrl subsystem [ 0.239184] [ 0.239734] ************************************************************* [ 0.241013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244011] ** ** [ 0.246010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.247007] ** ** [ 0.248008] ** This means that this kernel is built to expose internal ** [ 0.250012] ** IOMMU data structures, which may compromise security on ** [ 0.252028] ** your system. ** [ 0.254022] ** ** [ 0.256014] ** If you see this message and you are not debugging the ** [ 0.259016] ** kernel, report this immediately to your vendor! ** [ 0.260011] ** ** [ 0.261012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264012] ************************************************************* [ 0.265639] NET: Registered protocol family 16 [ 0.267505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276105] cpuidle: using governor menu [ 0.277508] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280080] PCI: Using configuration type 1 for base access [ 0.282125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.290230] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.296156] cryptd: max_cpu_qlen set to 1000 [ 0.301263] ACPI: Added _OSI(Module Device) [ 0.302015] ACPI: Added _OSI(Processor Device) [ 0.304016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309533] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315195] ACPI: Interpreter enabled [ 0.316049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.317009] ACPI: Using IOAPIC for interrupt routing [ 0.318101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.329146] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.331045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.334021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.343331] acpiphp: Slot [2] registered [ 0.345108] acpiphp: Slot [5] registered [ 0.347153] acpiphp: Slot [6] registered [ 0.348111] acpiphp: Slot [7] registered [ 0.350110] acpiphp: Slot [8] registered [ 0.351102] acpiphp: Slot [9] registered [ 0.353105] acpiphp: Slot [10] registered [ 0.354201] acpiphp: Slot [3] registered [ 0.356095] acpiphp: Slot [4] registered [ 0.357097] acpiphp: Slot [11] registered [ 0.359094] acpiphp: Slot [12] registered [ 0.360074] acpiphp: Slot [13] registered [ 0.361094] acpiphp: Slot [14] registered [ 0.363163] acpiphp: Slot [15] registered [ 0.364107] acpiphp: Slot [16] registered [ 0.366084] acpiphp: Slot [17] registered [ 0.367114] acpiphp: Slot [18] registered [ 0.368074] acpiphp: Slot [19] registered [ 0.370106] acpiphp: Slot [20] registered [ 0.371108] acpiphp: Slot [21] registered [ 0.373144] acpiphp: Slot [22] registered [ 0.374103] acpiphp: Slot [23] registered [ 0.376102] acpiphp: Slot [24] registered [ 0.378143] acpiphp: Slot [25] registered [ 0.379083] acpiphp: Slot [26] registered [ 0.380100] acpiphp: Slot [27] registered [ 0.382139] acpiphp: Slot [28] registered [ 0.383110] acpiphp: Slot [29] registered [ 0.385105] acpiphp: Slot [30] registered [ 0.387155] acpiphp: Slot [31] registered [ 0.388071] PCI host bridge to bus 0000:00 [ 0.390021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.392021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.401021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404824] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408104] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.416835] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.421596] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.424016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.426017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.428018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.431613] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.433837] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439752] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.445020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.460038] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.466015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473636] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.482016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.490017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522713] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.563023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.574358] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.585018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.592012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.622019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.648013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.674611] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.689016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.706029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.715000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.721014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.729017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.746016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.756173] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.760385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.761323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.763219] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.766176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.771113] iommu: Default domain type: Passthrough [ 0.772401] SCSI subsystem initialized [ 0.774117] ACPI: bus type USB registered [ 0.775080] usbcore: registered new interface driver usbfs [ 0.776000] usbcore: registered new interface driver hub [ 0.776000] usbcore: registered new device driver usb [ 0.778142] pps_core: LinuxPPS API ver. 1 registered [ 0.780011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782049] PTP clock support registered [ 0.784176] EDAC MC: Ver: 3.0.0 [ 0.785282] PCI: Using ACPI for IRQ routing [ 0.788075] NetLabel: Initializing [ 0.789015] NetLabel: domain hash size = 128 [ 0.790019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.793109] NetLabel: unlabeled traffic allowed by default [ 0.795202] vgaarb: loaded [ 0.797308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.799029] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.815969] clocksource: Switched to clocksource kvm-clock [ 0.927433] VFS: Disk quotas dquot_6.6.0 [ 0.929468] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.932916] *** VALIDATE ramfs *** [ 0.934272] *** VALIDATE hugetlbfs *** [ 0.935707] pnp: PnP ACPI init [ 0.938080] pnp: PnP ACPI: found 6 devices [ 0.955543] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.959155] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.961662] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.964039] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966981] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.969555] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.973118] NET: Registered protocol family 2 [ 0.976531] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.983477] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.987954] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.994028] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.997816] TCP: Hash tables configured (established 65536 bind 65536) [ 1.000467] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.004444] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009947] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012845] NET: Registered protocol family 1 [ 1.015912] RPC: Registered named UNIX socket transport module. [ 1.018240] RPC: Registered udp transport module. [ 1.020333] RPC: Registered tcp transport module. [ 1.022237] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.024624] NET: Registered protocol family 44 [ 1.026843] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.028952] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.031221] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.033187] PCI: CLS 0 bytes, default 64 [ 1.034358] Unpacking initramfs... [ 2.427563] debug: unmapping init [mem 0xffff9d343cc54000-0xffff9d343ffbffff] [ 2.431217] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.433898] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.437108] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.939606] Initialise system trusted keyrings [ 2.940869] Key type blacklist registered [ 2.942598] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.950815] zbud: loaded [ 2.954367] *** VALIDATE nfs *** [ 2.955888] *** VALIDATE nfs4 *** [ 2.957387] pstore: using deflate compression [ 2.961967] Platform Keyring initialized [ 3.079119] NET: Registered protocol family 38 [ 3.080779] Key type asymmetric registered [ 3.082260] Asymmetric key parser 'x509' registered [ 3.084211] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.087283] io scheduler mq-deadline registered [ 3.088949] io scheduler kyber registered [ 3.090634] io scheduler bfq registered [ 3.092418] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.095023] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.097487] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100281] ACPI: Power Button [PWRF] [ 3.105307] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.111365] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.124404] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.132257] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.150489] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.178952] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.204740] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.209622] Non-volatile memory driver v1.3 [ 3.211208] Linux agpgart interface v0.103 [ 3.244544] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.248113] vda: detected capacity change from 0 to 74702848 [ 3.263881] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.268120] vdb: detected capacity change from 0 to 1073741824 [ 3.286507] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.290276] vdc: detected capacity change from 0 to 2621440000 [ 3.305809] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308504] vdd: detected capacity change from 0 to 2621440000 [ 3.325150] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.327840] vde: detected capacity change from 0 to 4294967296 [ 3.344845] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347944] vdf: detected capacity change from 0 to 4294967296 [ 3.356385] libphy: Fixed MDIO Bus: probed [ 3.362383] usbcore: registered new interface driver usbserial_generic [ 3.365260] usbserial: USB Serial support registered for generic [ 3.367556] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.371744] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.373602] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.376295] mousedev: PS/2 mouse device common for all mice [ 3.379376] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.381655] rtc_cmos 00:05: RTC can wake from S4 [ 3.385774] rtc_cmos 00:05: registered as rtc0 [ 3.386348] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.387398] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.392654] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.393668] intel_pstate: CPU model not supported [ 3.399809] hid: raw HID events driver (C) Jiri Kosina [ 3.402086] usbcore: registered new interface driver usbhid [ 3.404114] usbhid: USB HID core driver [ 3.405696] drop_monitor: Initializing network drop monitor service [ 3.408135] Initializing XFRM netlink socket [ 3.409690] NET: Registered protocol family 10 [ 3.411978] Segment Routing with IPv6 [ 3.413540] NET: Registered protocol family 17 [ 3.415334] mpls_gso: MPLS GSO support [ 3.422741] RAS: Correctable Errors collector initialized. [ 3.425374] AVX version of gcm_enc/dec engaged. [ 3.427287] AES CTR mode by8 optimization enabled [ 3.495974] sched_clock: Marking stable (3495955639, 0)->(4471763733, -975808094) [ 3.498281] registered taskstats version 1 [ 3.499727] Loading compiled-in X.509 certificates [ 3.501765] zswap: loaded using pool lzo/zbud [ 3.524094] Key type big_key registered [ 3.534819] Key type encrypted registered [ 3.536351] ima: No TPM chip found, activating TPM-bypass! [ 3.538436] ima: Allocated hash algorithm: sha1 [ 3.539837] ima: No architecture policies found [ 3.542373] evm: Initialising EVM extended attributes: [ 3.543910] evm: security.selinux [ 3.544834] evm: security.ima [ 3.545669] evm: security.capability [ 3.546587] evm: HMAC attrs: 0x1 [ 3.548576] rtc_cmos 00:05: setting system clock to 2026-08-15 20:27:00 UTC (1786825620) [ 3.554122] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.556840] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.561234] Write protecting the kernel read-only data: 28672k [ 3.564335] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.566779] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.597054] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.604638] systemd[1]: Detected virtualization kvm. [ 3.606894] systemd[1]: Detected architecture x86-64. [ 3.610940] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.636264] systemd[1]: No hostname configured. [ 3.638190] systemd[1]: Set hostname to . [ 3.640293] random: systemd: uninitialized urandom read (16 bytes read) [ 3.643250] systemd[1]: Initializing machine ID from random generator. [ 3.690817] random: ln: uninitialized urandom read (6 bytes read) [ 3.783645] random: systemd: uninitialized urandom read (16 bytes read) [ 3.786475] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.790664] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.795647] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.398329] device-mapper: uevent: version 1.0.3 [ 4.402181] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 5.092190] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.132844] virtio_net virtio0 ens2: renamed from eth0 [ 5.162240] scsi host0: ata_piix [ 5.199441] scsi host1: ata_piix [ 5.208278] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.210648] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.876425] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.065378] random: crng init done [ 10.066509] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.631258] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.963653] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.612235] SELinux: Disabled at runtime. [ 14.712111] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.728834] systemd[1]: Detected virtualization kvm. [ 14.732301] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.872373] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.877839] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.897615] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.904682] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.911347] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.928046] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.939986] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ 16.096147] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 16.876318] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.536651] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.606454] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.782680] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.817546] EDAC sbridge: Ver: 1.1.2 [ 21.967453] Key type dns_resolver registered [ 22.483690] NFS: Registering the id_resolver key type [ 22.490600] Key type id_resolver registered [ 22.493932] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg614-server login: [ 39.441272] hrtimer: interrupt took 4265259 ns [ 68.414538] libcfs: loading out-of-tree module taints kernel. [ 68.456927] Key type ._llcrypt registered [ 68.459785] Key type .llcrypt registered [ 68.539657] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 85.519733] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 87.927943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.979760] alg: No test for adler32 (adler32-zlib) [ 89.526881] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 90.622081] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 92.423257] Key type lgssc registered [ 94.509419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.526711] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.374887] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 161.780133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.823985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.194716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.229908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.329186] Lustre: lustre-MDT0000: new disk, initializing [ 163.407304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.440248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.469851] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.589245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.720040] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.768281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.775304] Lustre: Skipped 1 previous similar message [ 177.938059] Lustre: lustre-MDT0001: new disk, initializing [ 178.033855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.060658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.069640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.614541] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.644613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.047821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.337619] Lustre: lustre-OST0000: new disk, initializing [ 194.343676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.354691] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.491862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.903296] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.248951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.258894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.356469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.968801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.134236] Lustre: lustre-OST0001: new disk, initializing [ 217.139251] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.142692] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.191074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.303906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.312293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.436499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.506195] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.119130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.997654] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.158046] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.174774] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing yml_node [ 258.001289] Lustre: DEBUG MARKER: Client: 2.17.57.5 [ 260.635675] Lustre: DEBUG MARKER: MDS: 2.17.57.5 [ 263.273964] Lustre: DEBUG MARKER: OSS: 2.17.57.5 [ 265.108924] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 16:31:19 EDT 2026 [ 281.692826] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 282.797972] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.514470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.517638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.527824] Lustre: Skipped 2 previous similar messages [ 300.535802] Lustre: Skipped 3 previous similar messages [ 303.407178] Lustre: server umount lustre-MDT0000 complete [ 306.345922] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825923 with bad export cookie 16840313391036490431 [ 306.348429] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.357639] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 310.752203] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.753569] LustreError: 8761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.766123] Lustre: Skipped 1 previous similar message [ 310.774082] LustreError: 8761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 310.786931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.782184] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.786024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.809111] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 311.834545] Lustre: Skipped 2 previous similar messages [ 312.755706] Lustre: server umount lustre-MDT0001 complete [ 316.221438] Lustre: server umount lustre-OST0000 complete [ 319.456368] Lustre: server umount lustre-OST0001 complete [ 325.282637] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 333.571881] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 365.539201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.760916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 365.798613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 365.881251] Lustre: lustre-MDT0000: new disk, initializing [ 366.006919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.073129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 371.203068] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 380.589521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.723190] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.777804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.782069] Lustre: Skipped 1 previous similar message [ 380.842164] Lustre: lustre-MDT0001: new disk, initializing [ 380.927862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.963428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.982158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 385.150502] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 393.766676] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.078629] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 405.527523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.897349] Lustre: lustre-OST0000: new disk, initializing [ 405.909369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 405.919511] Lustre: 17034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.090303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.900718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.920121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.050306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.410784] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 421.424893] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 425.733276] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 425.904193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.688736] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 429.845959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.634790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.644248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.652117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.670922] Lustre: Skipped 2 previous similar messages [ 433.687432] Lustre: Skipped 1 previous similar message [ 437.723843] Lustre: server umount lustre-OST0000 complete [ 442.337487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 442.354608] LustreError: Skipped 1 previous similar message [ 442.361035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.378525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 447.328716] Lustre: server umount lustre-MDT0000 complete [ 448.995781] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.017714] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 451.024850] LustreError: 17033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826067 with bad export cookie 16840313391036491964 [ 451.027544] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.038898] LustreError: 17033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 451.482821] Lustre: server umount lustre-MDT0001 complete [ 457.284658] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:34:31 (1786826071) [ 473.370239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.808745] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.878475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.225139] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 479.201880] LustreError: 19908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.052901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.967686] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 497.100756] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.246095] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.136836] Lustre: Failing over lustre-MDT0000 [ 504.539525] Lustre: server umount lustre-MDT0000 complete [ 506.849241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.858467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.879541] Lustre: Skipped 1 previous similar message [ 506.893792] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.915801] LustreError: 19907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 524.256256] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826125/real 1786826125] req@ffff9d34aea80000 x1873622552697856/t0(0) o400->MGC192.168.206.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786826141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 524.289592] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.307970] LustreError: 19904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.327175] LustreError: 19904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 524.732815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.496449] LustreError: 3637:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d3383da2d80 x1873622552700672/t0(0) o250->MGC192.168.206.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 538.398536] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 540.137841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 540.140971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 540.189720] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 547.105593] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 554.464194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.468125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.486579] Lustre: Skipped 2 previous similar messages [ 554.521609] Lustre: Skipped 3 previous similar messages [ 557.216327] Lustre: server umount lustre-MDT0000 complete [ 560.648313] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826177 with bad export cookie 16840313391036493378 [ 560.654789] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.663944] LustreError: 19889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 561.006247] Lustre: server umount lustre-MDT0001 complete [ 568.387241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.774821] LustreError: 23245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.787754] LustreError: 23245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 568.820165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.823649] Lustre: Skipped 2 previous similar messages [ 572.384568] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 581.501575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.356730] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 595.033853] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.261772] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.962361] Lustre: Failing over lustre-MDT0000 [ 602.157883] Lustre: server umount lustre-MDT0000 complete [ 602.592284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.610729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.007568] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826220/real 1786826220] req@ffff9d34bf135880 x1873622552727680/t0(0) o400->MGC192.168.206.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786826236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.032192] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.433278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.216830] LustreError: 3637:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d3383da3800 x1873622552730496/t0(0) o250->MGC192.168.206.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 629.658949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 634.279664] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 634.857283] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 634.866328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.874392] Lustre: Skipped 1 previous similar message [ 634.898881] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.655658] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 650.213345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.214668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.230778] Lustre: Skipped 1 previous similar message [ 650.235591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.263634] Lustre: Skipped 3 previous similar messages [ 653.505732] Lustre: server umount lustre-MDT0000 complete [ 656.355681] LustreError: 23246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.368438] LustreError: 23246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 656.426871] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826273 with bad export cookie 16840313391036494316 [ 656.430601] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.435560] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 656.710905] Lustre: server umount lustre-MDT0001 complete [ 664.150499] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:37:58 (1786826278) [ 677.996728] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 687.099447] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 719.650063] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 728.587665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.761887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 728.790224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 728.848978] Lustre: lustre-MDT0000: new disk, initializing [ 728.898821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.904324] Lustre: Skipped 2 previous similar messages [ 728.915432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 732.836777] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 742.291456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.419597] Lustre: 29619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 742.660242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 746.691655] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 754.941282] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 760.277978] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 766.007575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 766.249801] Lustre: 31588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 767.446612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 767.476809] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 767.514315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 767.526397] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 771.453231] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 780.102625] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 784.626393] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 803.299216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 803.306116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.310209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.318346] Lustre: Skipped 2 previous similar messages [ 803.330813] Lustre: Skipped 3 previous similar messages [ 805.158602] Lustre: server umount lustre-OST0000 complete [ 814.968784] Lustre: server umount lustre-MDT0000 complete [ 818.657415] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826435 with bad export cookie 16840313391036494722 [ 818.665383] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.665732] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.671078] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 819.238565] Lustre: server umount lustre-MDT0001 complete [ 829.387194] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 831.951324] Key type lgssc unregistered [ 832.369233] LNet: 33544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.395279] LNetError: 33544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.449891] LNet: Removed LNI 192.168.206.114@tcp [ 834.548144] Key type .llcrypt unregistered [ 834.550847] Key type ._llcrypt unregistered [ 857.002718] Key type ._llcrypt registered [ 857.006535] Key type .llcrypt registered [ 857.161979] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 858.021682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.077142] alg: No test for adler32 (adler32-zlib) [ 859.163789] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 859.382561] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 861.058094] Key type lgssc registered [ 861.999629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 868.918771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 868.934171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.275947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.549472] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 880.608266] LustreError: 34975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.632883] LustreError: 34975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 881.239372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.501039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 885.113279] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 891.063576] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 896.240451] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 902.742919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.057621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.173503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 909.015242] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 915.403408] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 915.421134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 927.758607] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:42:22 (1786826542) [ 929.271482] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 930.899871] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 936.223689] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:42:31 (1786826551) [ 943.049327] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:42:37 (1786826557) [ 943.841989] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 950.334245] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 950.339273] Lustre: Skipped 2 previous similar messages [ 951.652770] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:42:46 (1786826566) [ 954.629768] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 954.634657] Lustre: Skipped 3 previous similar messages [ 968.400072] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 968.405191] Lustre: Skipped 6 previous similar messages [ 970.488643] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:43:05 (1786826585) [ 972.335844] Lustre: *** cfs_fail_loc=131b, val=2*** [ 975.138568] Lustre: *** cfs_fail_loc=131b, val=2*** [ 981.867961] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:43:16 (1786826596) [ 985.895620] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 985.905645] Lustre: Skipped 5 previous similar messages [ 995.431636] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:43:29 (1786826609) [ 1008.608728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1008.616637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.630833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.704595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.708436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.721671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.742947] Lustre: Skipped 1 previous similar message [ 1014.100982] Lustre: server umount lustre-OST0000 complete [ 1017.826848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.835482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.842481] Lustre: Skipped 1 previous similar message [ 1017.854531] Lustre: Skipped 1 previous similar message [ 1022.947937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.965811] Lustre: Skipped 1 previous similar message [ 1023.738435] Lustre: server umount lustre-MDT0000 complete [ 1026.952401] LustreError: 34955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826643 with bad export cookie 14114401739114868617 [ 1026.953904] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.962199] LustreError: 34955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1027.254041] Lustre: server umount lustre-MDT0001 complete [ 1037.415977] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 1040.524458] Key type lgssc unregistered [ 1040.829461] LNet: 42823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.840832] LNetError: 42823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.878719] LNet: Removed LNI 192.168.206.114@tcp [ 1042.130326] Key type .llcrypt unregistered [ 1042.133030] Key type ._llcrypt unregistered [ 1061.039406] Key type ._llcrypt registered [ 1061.042609] Key type .llcrypt registered [ 1061.182444] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 1077.238161] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1077.885887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.957842] alg: No test for adler32 (adler32-zlib) [ 1078.966349] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1079.258670] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1080.895109] Key type lgssc registered [ 1081.830953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1111.692445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1111.714986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.934761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1112.957134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1113.035967] Lustre: lustre-MDT0000: new disk, initializing [ 1113.105071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.133066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1116.796721] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1126.026992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.149861] Lustre: 45980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1126.196103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1126.199745] Lustre: Skipped 1 previous similar message [ 1126.301249] Lustre: lustre-MDT0001: new disk, initializing [ 1126.413071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.461560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1126.474183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1130.485319] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1138.834593] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1143.460149] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.737987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.020507] Lustre: lustre-OST0000: new disk, initializing [ 1150.027928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1150.038558] Lustre: 47944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1150.122080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.617332] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1156.679638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1156.687627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1156.745847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1162.435647] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1166.362857] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1166.572630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1171.190920] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1171.456978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1177.571618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1177.575300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.576264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.652471] Lustre: Skipped 1 previous similar message [ 1179.786538] Lustre: server umount lustre-OST0000 complete [ 1187.808263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1187.814131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.818191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.818198] Lustre: Skipped 1 previous similar message [ 1187.863472] Lustre: Skipped 1 previous similar message [ 1192.929029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.961577] Lustre: Skipped 2 previous similar messages [ 1198.047151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1198.051736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.087910] Lustre: Skipped 1 previous similar message [ 1198.390157] Lustre: server umount lustre-MDT0000 complete [ 1201.954300] LustreError: 45972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826818 with bad export cookie 14250675204218157042 [ 1201.958371] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.964702] LustreError: 45972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1202.290873] Lustre: server umount lustre-MDT0001 complete [ 1211.320580] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:47:06 (1786826826) [ 1216.378696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.781845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.637191] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1222.117510] LustreError: 50405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.134222] LustreError: 50405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1226.224128] LustreError: 50406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.449293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.803371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.848510] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1237.982720] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.551475] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1248.584020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.966441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.283823] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1260.111377] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1268.726839] LustreError: 52934:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1269.500664] Lustre: Modifying parameter general.timeout=1 in log params [ 1271.444873] Lustre: Modifying parameter general.timeout=3 in log params [ 1271.450168] Lustre: Skipped 1 previous similar message [ 1274.060203] Lustre: Modifying parameter general.timeout=6 in log params [ 1274.062934] Lustre: Skipped 2 previous similar messages [ 1278.785432] Lustre: Modifying parameter general.timeout=12 in log params [ 1278.790347] Lustre: Skipped 5 previous similar messages [ 1287.167288] Lustre: Modifying parameter general.timeout=22 in log params [ 1287.172185] Lustre: Skipped 9 previous similar messages [ 1303.197273] Lustre: Modifying parameter general.timeout=42 in log params [ 1303.200604] Lustre: Skipped 19 previous similar messages [ 1335.483107] Lustre: Modifying parameter general.timeout=83 in log params [ 1335.492862] Lustre: Skipped 40 previous similar messages [ 1377.928056] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:49:52 (1786826992) [ 1380.152406] Lustre: 59864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1381.513834] Lustre: 59963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1384.417793] Lustre: 60160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1384.426443] Lustre: 60160:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1399.863193] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1399.869596] Lustre: Skipped 64 previous similar messages [ 1408.670983] Lustre: 61735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1408.680600] Lustre: 61735:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1408.696416] LustreError: 61735:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1411.972485] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:50:26 (1786827026) [ 1422.305990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.315497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.330969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.381759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.397308] Lustre: Skipped 1 previous similar message [ 1427.404751] Lustre: server umount lustre-MDT0000 complete [ 1430.497529] LustreError: 50406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.519695] LustreError: 50406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1430.768262] LustreError: 50386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827047 with bad export cookie 14250675204218157924 [ 1430.786202] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.792163] LustreError: 50386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1431.043483] Lustre: server umount lustre-MDT0001 complete [ 1444.309715] Lustre: server umount lustre-OST0000 complete [ 1464.628116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.951717] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1465.434344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.710513] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1480.151568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.244280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1480.274963] Lustre: 64260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1485.496404] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1492.226081] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1497.380371] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.824905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.041811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1504.272321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.278966] Lustre: Skipped 1 previous similar message [ 1508.879296] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1515.054316] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1517.004424] Lustre: 66827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1525.217922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1525.226930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1525.242049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1525.248431] Lustre: Skipped 3 previous similar messages [ 1528.550946] Lustre: server umount lustre-OST0000 complete [ 1531.872909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1531.883617] LustreError: Skipped 1 previous similar message [ 1535.968843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.974748] Lustre: Skipped 2 previous similar messages [ 1537.812914] Lustre: server umount lustre-MDT0000 complete [ 1540.841890] LustreError: 64252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827157 with bad export cookie 14250675204218160752 [ 1540.855924] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1540.859293] LustreError: 64252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1541.129335] Lustre: server umount lustre-MDT0001 complete [ 1549.432696] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 1551.685329] Key type lgssc unregistered [ 1551.952564] LNet: 68035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.960446] LNetError: 68035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.996273] LNet: Removed LNI 192.168.206.114@tcp [ 1552.918400] Key type .llcrypt unregistered [ 1552.925741] Key type ._llcrypt unregistered [ 1572.982685] Key type ._llcrypt registered [ 1572.987306] Key type .llcrypt registered [ 1573.084028] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1574.088749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.102515] alg: No test for adler32 (adler32-zlib) [ 1575.140239] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1575.338169] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1576.991175] Key type lgssc registered [ 1577.917742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.215602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.237425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.621976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.434595] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1595.879098] LustreError: 69427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.913568] LustreError: 69427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1596.164608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.697846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.532323] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1608.232968] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1613.842167] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1621.247275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1621.476648] LustreError: 71253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.869500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.531264] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1634.226343] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1640.491772] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1647.593978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1647.609967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.625364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.657988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.668320] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.684561] Lustre: Skipped 1 previous similar message [ 1653.420733] Lustre: server umount lustre-MDT0000 complete [ 1655.786165] LustreError: 69422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.808604] LustreError: 69422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1656.509480] LustreError: 72378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827273 with bad export cookie 15997608316686019518 [ 1656.521634] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.528893] LustreError: 72378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1656.808851] Lustre: server umount lustre-MDT0001 complete [ 1670.073649] Lustre: server umount lustre-OST0000 complete [ 1693.310366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.885229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1693.959490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1694.331372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.641714] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1707.193366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.331920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1707.361392] Lustre: 74296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1707.804937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1712.239802] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1719.011903] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.468732] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1730.500619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.677672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1731.055835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.838990] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1744.094965] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.517516] Lustre: 76870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1747.154724] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1751.008458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.016619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.027374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.029779] Lustre: Skipped 2 previous similar messages [ 1756.649857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.658380] Lustre: Skipped 1 previous similar message [ 1757.025629] Lustre: server umount lustre-OST0000 complete [ 1761.760921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.768807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.790381] Lustre: Skipped 2 previous similar messages [ 1761.813512] Lustre: Skipped 2 previous similar messages [ 1766.509159] Lustre: server umount lustre-MDT0000 complete [ 1766.881551] LustreError: 74307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.902374] LustreError: 74307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1770.154411] LustreError: 74289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827387 with bad export cookie 15997608316686020351 [ 1770.160195] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.164548] LustreError: 74289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1770.393372] Lustre: server umount lustre-MDT0001 complete [ 1782.591663] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 1786.338303] Key type lgssc unregistered [ 1786.799513] LNet: 78086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.820350] LNetError: 78086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.839768] LNet: Removed LNI 192.168.206.114@tcp [ 1788.188756] Key type .llcrypt unregistered [ 1788.191055] Key type ._llcrypt unregistered [ 1811.406393] Key type ._llcrypt registered [ 1811.408118] Key type .llcrypt registered [ 1811.543289] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 1812.632611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.650885] alg: No test for adler32 (adler32-zlib) [ 1813.675883] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1813.868786] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 1815.519160] Key type lgssc registered [ 1816.935996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1823.315922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1823.332422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.890124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.179361] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1834.471958] LustreError: 79502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.496412] LustreError: 79502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1836.835480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.525661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.457694] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1848.417153] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1853.476364] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.401396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.887092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.357773] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 1872.184275] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1878.388094] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1879.290744] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1881.193821] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1881.205672] Lustre: Skipped 1 previous similar message [ 1883.654973] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1883.662246] Lustre: Skipped 2 previous similar messages [ 1887.691688] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1887.696710] Lustre: Skipped 4 previous similar messages [ 1896.093545] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1896.100707] Lustre: Skipped 8 previous similar messages [ 1912.911239] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1912.916712] Lustre: Skipped 17 previous similar messages [ 1945.172390] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1945.178996] Lustre: Skipped 36 previous similar messages [ 1976.231678] Lustre: 87596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1979.048538] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:59:53 (1786827593) [ 1980.563889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1982.048377] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:59:56 (1786827596) [ 1986.016906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.026926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.067260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.040281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.044988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.048624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.075573] Lustre: Skipped 1 previous similar message [ 1992.161594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.166593] Lustre: Skipped 1 previous similar message [ 1997.289963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.297784] Lustre: Skipped 1 previous similar message [ 1999.327272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1999.456360] Lustre: server umount lustre-OST0000 complete [ 2006.496624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2006.503647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.512158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.247953] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2017.512866] Lustre: server umount lustre-MDT0000 complete [ 2017.762494] LustreError: 79502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.784145] LustreError: 79502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2021.306774] LustreError: 79484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827638 with bad export cookie 11141964926380744337 [ 2021.315355] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.321232] LustreError: 79484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.049155] Lustre: server umount lustre-MDT0001 complete [ 2032.039754] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 2035.241832] Key type lgssc unregistered [ 2035.520676] LNet: 88852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.531412] LNetError: 88852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.556463] LNet: Removed LNI 192.168.206.114@tcp [ 2036.254632] Key type .llcrypt unregistered [ 2036.256646] Key type ._llcrypt unregistered [ 2040.255594] Key type ._llcrypt registered [ 2040.257644] Key type .llcrypt registered [ 2040.338573] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2044.299368] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules [ 2045.496095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.551111] alg: No test for adler32 (adler32-zlib) [ 2046.546729] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2046.702748] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 2048.312476] Key type lgssc registered [ 2049.274675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.622237] LustreError: 90067:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2066.762297] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2072.449333] LustreError: 90067:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2075.640606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2075.661475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.163220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.147730] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2087.392375] LustreError: 91091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.410607] LustreError: 91091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2088.410283] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:01:43 (1786827703) [ 2092.421110] LustreError: 91123:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2092.427861] LustreError: 91123:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2092.437754] LustreError: 91123:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2092.801404] Lustre: server umount lustre-MDT0000 complete [ 2102.940388] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 2105.512239] Key type lgssc unregistered [ 2105.853229] LNet: 92617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.871430] LNetError: 92617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.889784] LNet: Removed LNI 192.168.206.114@tcp [ 2106.642567] Key type .llcrypt unregistered [ 2106.648087] Key type ._llcrypt unregistered [ 2122.289057] Key type ._llcrypt registered [ 2122.291340] Key type .llcrypt registered [ 2122.382911] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2123.024181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.036165] alg: No test for adler32 (adler32-zlib) [ 2123.999696] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2124.191439] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 2125.847185] Key type lgssc registered [ 2126.781751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2133.148921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2133.178892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.698348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.827839] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2144.237866] LustreError: 93836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.263791] LustreError: 93836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2146.476736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.039706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.148877] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2157.294636] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2161.782661] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2167.277719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.703730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.877903] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2179.120666] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2220.801985] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:03:55 (1786827835) [ 2221.939989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2223.478373] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:03:58 (1786827838) [ 2228.709643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2228.722573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.735987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.274858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.276855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.293340] Lustre: Skipped 1 previous similar message [ 2233.328286] Lustre: server umount lustre-MDT0000 complete [ 2236.386093] LustreError: 94567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.405928] LustreError: 94567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2236.493540] LustreError: 95688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827853 with bad export cookie 18104616847680714546 [ 2236.501023] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.505624] LustreError: 95688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2236.799343] Lustre: server umount lustre-MDT0001 complete [ 2250.748601] Lustre: server umount lustre-OST0000 complete [ 2262.541412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.161435] LustreError: 98107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.433314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.582410] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2268.642276] LustreError: 98108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.766126] LustreError: 98107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.378220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.138167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2280.670894] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2287.980410] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2292.804134] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2304.303545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2304.511285] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2304.516423] LustreError: 98096:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2304.553642] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2304.561847] LustreError: 100190:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2304.566378] LustreError: 100190:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2304.571037] LustreError: 100190:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2304.622332] Lustre: server umount lustre-OST0000 complete [ 2304.625241] LustreError: 100190:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2312.684087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.873973] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2312.883658] LustreError: 98095:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2312.890968] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2312.896117] LustreError: 100651:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2312.900836] LustreError: 100651:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2312.905257] LustreError: 100651:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2312.953735] Lustre: server umount lustre-OST0000 complete [ 2312.956869] LustreError: 100651:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2321.577042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2321.856986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2321.870590] Lustre: Found index 0 for lustre-OST0000, updating log [ 2321.877615] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2322.007493] Lustre: lustre-OST0000: new disk, initializing [ 2322.013086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2322.022426] Lustre: 101187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2322.256066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.708978] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2333.528357] Lustre: Failing over lustre-OST0000 [ 2333.570870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -19 [ 2333.575988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.589354] LustreError: 98127:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -19 [ 2333.591190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.606209] Lustre: Skipped 2 previous similar messages [ 2338.786550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.737812] Lustre: server umount lustre-OST0000 complete [ 2343.906791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.912650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.925860] Lustre: Skipped 1 previous similar message [ 2343.936731] Lustre: Skipped 1 previous similar message [ 2356.193735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2356.200161] Lustre: Skipped 5 previous similar messages [ 2358.240315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2358.563815] Lustre: server umount lustre-MDT0000 complete [ 2361.315091] LustreError: 98108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.329290] LustreError: 98108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2362.961027] LustreError: 98089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827979 with bad export cookie 18104616847680717283 [ 2362.963700] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.968212] LustreError: 98089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2363.223373] Lustre: server umount lustre-MDT0001 complete [ 2372.362951] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:06:26 (1786827986) [ 2392.693320] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2401.767512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.147693] LustreError: 103534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.220357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.949960] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2414.382582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.910874] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2422.291278] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2428.800892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.511441] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2435.113252] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2439.715439] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2439.729036] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2439.743196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2443.521437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2443.580294] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2443.669416] Lustre: lustre-OST0001: new disk, initializing [ 2443.673344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2443.677612] Lustre: 105854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2443.928697] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2443.959980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2443.975812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.983550] Lustre: Skipped 2 previous similar messages [ 2444.691226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2444.706324] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2444.803824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2447.859436] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:01 [ 2447.886599] Lustre: Skipped 1 previous similar message [ 2447.899416] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2447.900762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2449.843416] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2456.363179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2466.629241] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2471.906632] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.915971] LustreError: Skipped 2 previous similar messages [ 2471.920967] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.927211] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2476.228759] Lustre: server umount lustre-MDT0001 complete [ 2477.447590] LustreError: 106592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.474538] LustreError: 106592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2485.382044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.483306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2485.486930] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2485.530544] Lustre: 103523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2487.603797] LustreError: 107442:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2487.612494] LustreError: 107442:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2487.630396] LustreError: 103526:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2487.752664] LustreError: 107452:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.114@tcp: cfg command failed: rc = -17 [ 2487.771140] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2487.771140] [ 2490.019310] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2491.371830] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2498.607852] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:08:33 (1786828113) [ 2513.680778] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2528.402245] Lustre: 109467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2549.810742] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2552.938729] Lustre: 110602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2634.208458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2634.219720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.241663] Lustre: Skipped 2 previous similar messages [ 2634.250470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.254127] Lustre: Skipped 4 previous similar messages [ 2638.069109] Lustre: server umount lustre-MDT0000 complete [ 2639.840853] LustreError: 106592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.858971] LustreError: 106592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2641.141756] LustreError: 103516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828258 with bad export cookie 18104616847680718221 [ 2641.147792] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.156279] LustreError: 103516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2641.436101] Lustre: server umount lustre-MDT0001 complete [ 2654.265469] Lustre: server umount lustre-OST0000 complete [ 2666.464699] Lustre: server umount lustre-OST0001 complete [ 2671.720984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2681.766673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2690.194853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2699.627816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2716.802228] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2726.564872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.599574] Lustre: lustre-MDT0000: reset Object Index mappings [ 2727.229653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.233872] Lustre: Skipped 1 previous similar message [ 2731.195554] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2738.399224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.427059] Lustre: lustre-MDT0001: reset Object Index mappings [ 2743.236627] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2745.918845] Lustre: 115704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2752.089946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.133978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2758.173787] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2763.273640] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2765.365715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.935725] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2770.942449] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2771.487661] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2778.504672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2781.539542] Lustre: 117581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2821.951080] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:13:56 (1786828436) [ 2851.300813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.312769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.325833] Lustre: Skipped 3 previous similar messages [ 2851.332755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.337813] Lustre: Skipped 3 previous similar messages [ 2856.155453] Lustre: server umount lustre-MDT0000 complete [ 2857.955571] LustreError: 114554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.983243] LustreError: 114554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2859.186086] LustreError: 114538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828476 with bad export cookie 18104616847680765093 [ 2859.196621] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2859.209693] LustreError: 114538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2865.668887] Lustre: server umount lustre-MDT0001 complete [ 2874.628965] Lustre: server umount lustre-OST0000 complete [ 2884.540966] Lustre: server umount lustre-OST0001 complete [ 2890.076789] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 2898.927261] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2931.147535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.411314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2931.445585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2931.512361] Lustre: lustre-MDT0000: new disk, initializing [ 2931.601525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.606068] Lustre: Skipped 3 previous similar messages [ 2931.621382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2935.676300] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2941.766035] Lustre: server umount lustre-MDT0000 complete [ 2949.529065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.755585] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2949.767975] Lustre: 121843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2949.779946] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2953.575247] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2957.861211] Lustre: server umount lustre-MDT0000 complete [ 2964.805755] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:16:19 (1786828579) [ 2966.087678] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2967.434665] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:16:22 (1786828582) [ 2980.411424] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 2988.074888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.087575] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 2999.531064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.679807] Lustre: 123892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.735765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2999.739893] Lustre: Skipped 1 previous similar message [ 2999.811329] Lustre: lustre-MDT0001: new disk, initializing [ 2999.955238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2999.969635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3003.734734] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3008.058631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3014.158594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.479734] Lustre: lustre-OST0000: new disk, initializing [ 3014.484078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3014.492077] Lustre: 125469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.661507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3015.682587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3015.842687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3021.709562] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3032.401369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.603943] Lustre: lustre-OST0001: new disk, initializing [ 3032.606933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3032.615102] Lustre: 126339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3034.343805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3034.350759] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3034.468943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3038.123497] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3046.189206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.995243] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3380.394334] Lustre: 132742:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3380.406739] Lustre: 132742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3380.417504] Lustre: 132742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3380.422744] Lustre: 132742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3380.429443] Lustre: 132742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3380.434637] Lustre: 132742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3635.729543] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:27:30 (1786829250) [ 3650.528304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3650.540171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.555717] Lustre: Skipped 5 previous similar messages [ 3650.564290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.579215] Lustre: Skipped 8 previous similar messages [ 3656.225618] Lustre: server umount lustre-MDT0000 complete [ 3659.233207] LustreError: 125463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.243459] LustreError: 125463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3659.823645] LustreError: 132764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829276 with bad export cookie 18104616847680830879 [ 3659.829535] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3659.836582] LustreError: 132764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3664.354365] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.372610] Lustre: Skipped 4 previous similar messages [ 3666.388712] Lustre: server umount lustre-MDT0001 complete [ 3684.319118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3684.425635] Lustre: server umount lustre-OST0000 complete [ 3693.484754] Lustre: server umount lustre-OST0001 complete [ 3700.919679] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 3712.238889] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 3743.839088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.120345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3744.150132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3744.226559] Lustre: lustre-MDT0000: new disk, initializing [ 3744.310447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.315294] Lustre: Skipped 5 previous similar messages [ 3744.325172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3748.088550] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3757.515665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.611523] Lustre: 135783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3757.668988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3757.673823] Lustre: Skipped 1 previous similar message [ 3757.766714] Lustre: lustre-MDT0001: new disk, initializing [ 3757.924507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3757.937153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3761.411349] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3769.339389] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.867259] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3779.749621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.930790] Lustre: lustre-OST0000: new disk, initializing [ 3779.936080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3779.941141] Lustre: 137752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3781.948618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3781.966571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3782.072072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3785.289432] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 3794.205128] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3803.838963] Lustre: lustre-MDD0000: changelog on [ 3806.954348] Lustre: lustre-MDD0001: changelog on [ 5357.081909] Lustre: lustre-MDD0001: changelog off [ 5358.430969] Lustre: 141083:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5359.182832] Lustre: lustre-MDD0000: changelog off [ 5369.317032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5369.319200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.337273] LustreError: Skipped 1 previous similar message [ 5369.338899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5369.352330] Lustre: Skipped 1 previous similar message [ 5369.366828] Lustre: Skipped 7 previous similar messages [ 5371.261331] Lustre: server umount lustre-OST0000 complete [ 5375.969321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5375.978170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.997455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.556531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.673899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5384.678145] Lustre: Skipped 3 previous similar messages [ 5389.279106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5389.485570] Lustre: server umount lustre-MDT0000 complete [ 5389.792761] LustreError: 135796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.806716] LustreError: 135796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5392.373203] LustreError: 135778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831009 with bad export cookie 18104616847681647051 [ 5392.386233] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.392319] LustreError: 135778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5392.679892] Lustre: server umount lustre-MDT0001 complete [ 5402.247573] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 5404.986770] Key type lgssc unregistered [ 5405.275868] LNet: 146840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5405.281836] LNetError: 146840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5405.307564] LNet: Removed LNI 192.168.206.114@tcp [ 5406.177363] Key type .llcrypt unregistered [ 5406.182676] Key type ._llcrypt unregistered [ 5425.679593] Key type ._llcrypt registered [ 5425.684437] Key type .llcrypt registered [ 5425.844676] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 5443.001595] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 5444.339311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5444.498969] alg: No test for adler32 (adler32-zlib) [ 5445.522428] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5445.752743] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 5447.471236] Key type lgssc registered [ 5448.532166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5479.976580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5479.989235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.219051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5481.273717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5481.395641] Lustre: lustre-MDT0000: new disk, initializing [ 5481.503657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.524384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5485.234339] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5494.305300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.382340] Lustre: 150056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5494.437669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5494.444675] Lustre: Skipped 1 previous similar message [ 5494.520524] Lustre: lustre-MDT0001: new disk, initializing [ 5494.602731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5494.635631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5494.643661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5498.065156] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5505.205742] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5509.572429] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5514.511555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.793727] Lustre: lustre-OST0000: new disk, initializing [ 5514.802074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5514.808328] Lustre: 152027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5514.906974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.785875] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5524.032175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5524.039613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5524.105089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5527.761897] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.335350] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_module obdecho/obdecho [ 5535.095939] LustreError: 153031:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5535.111040] LustreError: 153031:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5536.809456] LustreError: 153127:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5550.053839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.060984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5550.071964] Lustre: Skipped 2 previous similar messages [ 5550.076332] Lustre: Skipped 1 previous similar message [ 5552.329700] Lustre: server umount lustre-MDT0000 complete [ 5554.996192] LustreError: 150049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831171 with bad export cookie 9176905995740084659 [ 5554.998803] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.010206] LustreError: 150049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5555.170476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.292244] Lustre: server umount lustre-MDT0001 complete [ 5564.883598] Lustre: server umount lustre-OST0000 complete [ 5581.811839] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 5591.352566] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 5619.208672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.416139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5619.444447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5619.506797] Lustre: lustre-MDT0000: new disk, initializing [ 5619.563780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.584016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5622.981884] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5630.972289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.053373] Lustre: 156842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5631.087566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5631.090910] Lustre: Skipped 1 previous similar message [ 5631.145874] Lustre: lustre-MDT0001: new disk, initializing [ 5631.216444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5631.252309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5631.262628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5634.445344] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5642.225919] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.650576] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5651.808726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5652.008920] Lustre: lustre-OST0000: new disk, initializing [ 5652.012321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5652.017923] Lustre: 158817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5652.079400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.818251] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5661.260704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5661.273389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5661.392132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5664.503636] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5667.945753] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5668.130415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5671.704827] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5671.850970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5677.030567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.030597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.032764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.032771] Lustre: Skipped 1 previous similar message [ 5677.038824] LustreError: Skipped 1 previous similar message [ 5677.051438] Lustre: Skipped 1 previous similar message [ 5679.368716] Lustre: server umount lustre-OST0000 complete [ 5682.150394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.154652] LustreError: 148279:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000098f84e0e x1873628167740544/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5682.165515] Lustre: Skipped 1 previous similar message [ 5682.197349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.207467] Lustre: Skipped 2 previous similar messages [ 5687.270537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.274125] Lustre: Skipped 1 previous similar message [ 5688.460235] Lustre: server umount lustre-MDT0000 complete [ 5691.719424] LustreError: 157634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831308 with bad export cookie 9176905995740085618 [ 5691.724152] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5691.732875] LustreError: 157634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5692.155373] Lustre: server umount lustre-MDT0001 complete [ 5695.373973] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:01:49 (1786831309) [ 5696.697190] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5698.156590] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:01:53 (1786831313) [ 5704.602349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.227542] LustreError: 161227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.247648] LustreError: 161227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5705.342647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.777392] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5710.820343] LustreError: 161228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5715.939431] LustreError: 161227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.024668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.157935] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5728.432801] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5732.498901] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5737.779567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5738.029732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.035164] Lustre: Skipped 1 previous similar message [ 5743.429981] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5749.337349] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5758.686895] LustreError: 163807:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x3899d3af (00000000bd00e242) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5759.870155] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.14@tcp (stopping) [ 5760.488792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.500622] Lustre: Skipped 2 previous similar messages [ 5764.945439] Lustre: server umount lustre-MDT0000 complete [ 5764.972743] LustreError: 161222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.018028] LustreError: 161207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831384 with bad export cookie 9176905995740086500 [ 5768.020234] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5768.029133] LustreError: 161207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5770.099045] LustreError: 163064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.113899] LustreError: 163064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5770.118133] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.14@tcp (stopping) [ 5770.127199] Lustre: Skipped 3 previous similar messages [ 5770.725426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.514625] Lustre: server umount lustre-MDT0001 complete [ 5780.117773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5790.913122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.097170] LustreError: 165069:0:(llog.c:1655:llog_backup()) MGC192.168.206.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5791.105372] Lustre: 165069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5797.349949] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.381439] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5817.193432] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.212570] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5852.024653] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.041384] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 5852.129129] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5852.139888] Lustre: 165069:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5893.115179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.868884] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5898.217848] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5903.757554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.585169] LustreError: 148279:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000edc023c9 x1873628167787776/t0(0) o250->MGC192.168.206.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5917.737536] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 5919.207400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5923.911596] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.257693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5929.581996] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5930.753592] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5948.115628] Lustre: server umount lustre-MDT0000 complete [ 5949.922642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.925696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.930661] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.930671] LustreError: 165074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 5949.979712] Lustre: Skipped 2 previous similar messages [ 5951.082265] LustreError: 163067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831568 with bad export cookie 9176905995740092954 [ 5951.088859] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.094353] LustreError: 163067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5951.355392] Lustre: server umount lustre-MDT0001 complete [ 5965.025601] Lustre: server umount lustre-OST0000 complete [ 5981.416942] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 5990.631893] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6022.132642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.300548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6022.341706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6022.411262] Lustre: lustre-MDT0000: new disk, initializing [ 6022.472604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.478255] Lustre: Skipped 1 previous similar message [ 6022.490130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6025.546901] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6032.964817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.051947] Lustre: 170171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6033.301570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6036.584610] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6044.498572] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6049.027597] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6054.600269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6054.850329] Lustre: lustre-OST0000: new disk, initializing [ 6054.853297] Lustre: Skipped 1 previous similar message [ 6054.856549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6054.858801] Lustre: Skipped 2 previous similar messages [ 6054.862208] Lustre: 172136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6059.883718] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6064.181566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6064.187954] Lustre: Skipped 1 previous similar message [ 6064.196250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6064.288498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6067.412882] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6070.904669] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6071.076174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6074.715457] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6074.893947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6079.974030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.974196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.980052] LustreError: Skipped 1 previous similar message [ 6079.982547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.992330] Lustre: Skipped 1 previous similar message [ 6080.010646] Lustre: Skipped 3 previous similar messages [ 6082.376347] Lustre: server umount lustre-OST0000 complete [ 6089.700261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6091.707643] Lustre: server umount lustre-MDT0000 complete [ 6094.601507] LustreError: 170965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831711 with bad export cookie 9176905995740109789 [ 6094.604461] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.608693] LustreError: 170965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6094.821555] Lustre: server umount lustre-MDT0001 complete [ 6101.114231] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:08:35 (1786831715) [ 6105.673342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.002280] LustreError: 174547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.011790] LustreError: 174547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6108.535851] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6114.704616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.132895] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6124.400173] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.363500] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.316677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6142.455775] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6148.081745] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.606737] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:09:37 (1786831777) [ 6170.605304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6170.615184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.635412] Lustre: Skipped 2 previous similar messages [ 6170.640609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6170.647143] Lustre: Skipped 3 previous similar messages [ 6173.141896] Lustre: server umount lustre-OST0000 complete [ 6182.487824] Lustre: server umount lustre-MDT0000 complete [ 6185.702198] LustreError: 174529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831802 with bad export cookie 9176905995740110671 [ 6185.708583] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.713769] LustreError: 174529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6186.008766] Lustre: server umount lustre-MDT0001 complete [ 6195.178947] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 6197.545273] Key type lgssc unregistered [ 6197.799955] LNet: 178274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6197.809402] LNetError: 178274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6197.827434] LNet: Removed LNI 192.168.206.114@tcp [ 6198.428483] Key type .llcrypt unregistered [ 6198.430118] Key type ._llcrypt unregistered [ 6213.842798] Key type ._llcrypt registered [ 6213.844144] Key type .llcrypt registered [ 6213.931369] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6215.016438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.040152] alg: No test for adler32 (adler32-zlib) [ 6216.073925] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6216.257919] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 6217.936353] Key type lgssc registered [ 6218.866730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6225.827184] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6225.866920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.439489] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6246.559566] LustreError: 179519:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.114@tcp: failed processing log, type 4: rc = -110 [ 6251.680964] LustreError: 179519:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6251.689982] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6251.692924] LustreError: MGC192.168.206.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6251.699106] LustreError: 179519:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6251.703390] LustreError: 179519:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6251.715280] LustreError: 179519:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6251.771255] Lustre: server umount lustre-OST0000 complete [ 6251.773658] LustreError: 179519:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6258.758342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.147734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6263.838672] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6270.452301] LustreError: 180019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.487038] LustreError: 180019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6271.860558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.156354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6276.712823] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6283.125439] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6287.796829] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6294.183952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.423096] LustreError: 181834:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6294.432474] Lustre: 181834:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6294.594942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.476779] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6306.939219] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6310.885892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.898566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6310.904662] Lustre: Skipped 2 previous similar messages [ 6310.926095] Lustre: Skipped 2 previous similar messages [ 6316.004386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.011613] Lustre: Skipped 2 previous similar messages [ 6321.125451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6321.131393] Lustre: Skipped 2 previous similar messages [ 6323.169430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6323.356792] Lustre: server umount lustre-MDT0000 complete [ 6326.240501] LustreError: 180747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.279226] LustreError: 180747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6326.448838] LustreError: 181861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831943 with bad export cookie 3790631537269663075 [ 6326.453180] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6326.455982] LustreError: 181861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6326.667371] Lustre: server umount lustre-MDT0001 complete [ 6340.213703] Lustre: server umount lustre-OST0000 complete [ 6347.630465] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6358.342438] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6387.564468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.820826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6387.845231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6387.922631] Lustre: lustre-MDT0000: new disk, initializing [ 6387.990488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6388.002689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6391.567260] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6400.408370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.508270] Lustre: 185359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6400.555318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6400.560392] Lustre: Skipped 1 previous similar message [ 6400.625821] Lustre: lustre-MDT0001: new disk, initializing [ 6400.700248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6400.738087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6400.756055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6404.675499] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6412.937205] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6417.489680] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6422.791186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.991620] Lustre: lustre-OST0000: new disk, initializing [ 6422.994967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6423.002694] Lustre: 187328:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6423.089348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6428.289944] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6433.394098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6433.418374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6433.543897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6436.832951] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6440.671479] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6440.871574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6445.487637] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6445.726989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6449.128930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6449.129460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6449.135221] LustreError: Skipped 1 previous similar message [ 6449.144608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6449.153833] Lustre: Skipped 1 previous similar message [ 6453.540848] Lustre: server umount lustre-OST0000 complete [ 6457.317269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6457.327668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.349561] Lustre: Skipped 1 previous similar message [ 6457.354748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.384378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.943526] Lustre: server umount lustre-MDT0000 complete [ 6464.483923] LustreError: 188136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.508653] LustreError: 188136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6466.195190] LustreError: 185351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832083 with bad export cookie 3790631537269663789 [ 6466.200373] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.205494] LustreError: 185351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6466.384121] Lustre: server umount lustre-MDT0001 complete [ 6473.687467] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:14:48 (1786832088) [ 6488.242600] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 6490.755258] Key type lgssc unregistered [ 6491.041794] LNet: 190274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6491.051636] LNetError: 190274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6491.063247] LNet: Removed LNI 192.168.206.114@tcp [ 6491.898115] Key type .llcrypt unregistered [ 6491.900333] Key type ._llcrypt unregistered [ 6508.113686] Key type ._llcrypt registered [ 6508.116570] Key type .llcrypt registered [ 6508.208587] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6509.185701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6509.212659] alg: No test for adler32 (adler32-zlib) [ 6510.283167] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6510.567968] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 6512.215154] Key type lgssc registered [ 6513.173027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6520.131780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6520.164729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.626985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.442693] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6532.081935] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.100080] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6532.296020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.692253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6537.240301] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6543.473680] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6548.721441] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.427823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.613912] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6555.616406] LustreError: 193356:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6555.625984] Lustre: 193356:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6555.727933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.934264] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6567.604369] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6573.541306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.547926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.553299] Lustre: Skipped 1 previous similar message [ 6573.554265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.577891] Lustre: Skipped 2 previous similar messages [ 6575.819699] Lustre: server umount lustre-MDT0000 complete [ 6578.657507] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.679510] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6578.760540] LustreError: 191475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832195 with bad export cookie 8697171621157994333 [ 6578.770281] LustreError: 191475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6578.770660] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.104869] Lustre: server umount lustre-MDT0001 complete [ 6593.077768] Lustre: server umount lustre-OST0000 complete [ 6601.428290] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6610.169431] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6639.029886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.196996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6639.214910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6639.261360] Lustre: lustre-MDT0000: new disk, initializing [ 6639.322445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.334918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6642.121955] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6649.654549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.737842] Lustre: 196878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6649.782693] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6649.784814] Lustre: Skipped 1 previous similar message [ 6649.786711] LustreError: 197662:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6649.790747] LustreError: 197662:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6649.795925] Lustre: 197662:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6649.815556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6649.821132] Lustre: Skipped 1 previous similar message [ 6649.885912] Lustre: lustre-MDT0001: new disk, initializing [ 6649.957408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6650.044705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6650.059963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6653.529657] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6660.757614] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6664.803289] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6669.684757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.863338] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6669.870602] Lustre: Skipped 1 previous similar message [ 6669.873313] LustreError: 198822:0:(llog.c:1696:llog_backup()) MGC192.168.206.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6669.879811] LustreError: 198822:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6669.886170] Lustre: 198822:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6669.893053] Lustre: 198822:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6669.914380] Lustre: lustre-OST0000: new disk, initializing [ 6669.917109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6669.922530] Lustre: 198848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6670.001726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.475393] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6678.584676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6678.596760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6678.668258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6681.543171] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.256485] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6685.401845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6688.556501] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6688.721458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6694.377598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.383542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6694.384282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.394409] Lustre: Skipped 2 previous similar messages [ 6696.242321] Lustre: server umount lustre-OST0000 complete [ 6699.490893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6699.498126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6699.500869] Lustre: Skipped 1 previous similar message [ 6699.504688] Lustre: Skipped 1 previous similar message [ 6704.609939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.728707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.731819] Lustre: Skipped 2 previous similar messages [ 6713.311101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6713.454476] Lustre: server umount lustre-MDT0000 complete [ 6714.848314] LustreError: 196891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6714.864354] LustreError: 196891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6716.517988] LustreError: 196872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832333 with bad export cookie 8697171621157995054 [ 6716.526751] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.529682] LustreError: 196872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6716.874557] Lustre: server umount lustre-MDT0001 complete [ 6722.657397] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:18:57 (1786832337) [ 6735.270295] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6743.161699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.485751] LustreError: 201971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.533333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.111262] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6748.640095] LustreError: 201972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.317139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.469590] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6758.273819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6762.387486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.118702] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6773.480984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.637358] Lustre: lustre-OST0001: new disk, initializing [ 6773.640287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6773.644064] Lustre: 204278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6777.722842] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6779.965482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6779.977545] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6780.052253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6785.842424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6791.627844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6793.169499] Lustre: DEBUG MARKER: ADD OST3 [ 6796.887245] loop: module loaded [ 6796.919080] loop0: detected capacity change from 0 to 204800000 [ 6797.124428] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6797.131636] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6797.143466] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6803.428079] loop0: detected capacity change from 0 to 204800000 [ 6804.418766] Lustre: DEBUG MARKER: STOP OST3 [ 6810.780405] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.952596] Lustre: lustre-OST0003: new disk, initializing [ 6810.972762] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6810.989467] Lustre: 206301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6811.157398] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6811.170034] Lustre: Skipped 3 previous similar messages [ 6812.029771] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6812.169682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6812.181052] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6812.227569] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6814.052802] LustreError: 206432:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6814.606283] Lustre: Failing over lustre-OST0003 [ 6814.675700] Lustre: server umount lustre-OST0003 complete [ 6817.255716] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.258585] LustreError: 203455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.277407] Lustre: Skipped 1 previous similar message [ 6818.045222] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6825.709269] Lustre: DEBUG MARKER: START OST3 again [ 6830.588247] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.802762] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6831.990942] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6832.057836] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6832.057879] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6835.167920] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6853.923570] Lustre: Failing over lustre-OST0003 [ 6854.005675] Lustre: server umount lustre-OST0003 complete [ 6856.682081] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.690739] LustreError: 204286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.692653] Lustre: Skipped 1 previous similar message [ 6856.716097] LustreError: 204286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6871.527219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6871.540441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.555963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.691848] Lustre: server umount lustre-MDT0000 complete [ 6878.385506] LustreError: 201953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832495 with bad export cookie 8697171621157995936 [ 6878.388465] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6878.392665] LustreError: 201953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6884.747559] Lustre: server umount lustre-MDT0001 complete [ 6888.117802] Lustre: server umount lustre-OST0000 complete [ 6891.156612] Lustre: server umount lustre-OST0001 complete [ 6897.106618] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 6903.778683] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 6932.139662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.353769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6932.386103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6932.476374] Lustre: lustre-MDT0000: new disk, initializing [ 6932.669980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.677341] Lustre: Skipped 1 previous similar message [ 6932.690995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6936.756449] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6945.592659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.707691] Lustre: 211733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6945.955101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6949.668339] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6957.663239] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6962.138595] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6967.668195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.917526] Lustre: lustre-OST0000: new disk, initializing [ 6967.921943] Lustre: Skipped 1 previous similar message [ 6967.926782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6967.930983] Lustre: Skipped 2 previous similar messages [ 6967.936227] Lustre: 213700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6969.034297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6969.047813] Lustre: Skipped 1 previous similar message [ 6969.053595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6969.140970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6973.048483] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 6980.628437] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6984.887437] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6985.150391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6989.497868] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6989.708248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6994.919412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.920883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.934125] LustreError: Skipped 1 previous similar message [ 6994.939577] Lustre: Skipped 5 previous similar messages [ 6994.940596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6994.940601] Lustre: Skipped 8 previous similar messages [ 6997.778534] Lustre: server umount lustre-OST0000 complete [ 7002.080406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7007.523938] Lustre: server umount lustre-MDT0000 complete [ 7010.274896] LustreError: 211741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.296270] LustreError: 211741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7010.494395] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832627 with bad export cookie 8697171621157998232 [ 7010.505476] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7010.515919] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7010.783795] Lustre: server umount lustre-MDT0001 complete [ 7013.917491] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:23:48 (1786832628) [ 7027.037314] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7034.569899] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7062.220738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.483914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7062.515981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7062.596616] Lustre: lustre-MDT0000: new disk, initializing [ 7062.674826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.680865] Lustre: Skipped 2 previous similar messages [ 7062.696441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7066.041680] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7075.066746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.202765] Lustre: 218310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7075.493638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7079.374332] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7087.182619] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7092.089629] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7097.977555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.258225] Lustre: 220278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7099.409516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7102.540609] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7110.032734] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.568292] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7114.796455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7120.257424] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7120.422883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7124.961108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.963902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.975138] LustreError: Skipped 1 previous similar message [ 7124.976138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7124.976143] Lustre: Skipped 4 previous similar messages [ 7125.025670] Lustre: Skipped 4 previous similar messages [ 7128.588551] Lustre: server umount lustre-OST0000 complete [ 7137.495909] Lustre: server umount lustre-MDT0000 complete [ 7140.324653] LustreError: 218317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.342039] LustreError: 218317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7140.446597] LustreError: 218302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832757 with bad export cookie 8697171621157999114 [ 7140.456980] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7140.460119] LustreError: 218302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7140.798758] Lustre: server umount lustre-MDT0001 complete [ 7147.617707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.504754] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7157.995881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.685364] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7167.358284] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7171.516490] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7176.384058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.590848] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7185.707255] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.417691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.425883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7208.432421] Lustre: Skipped 3 previous similar messages [ 7208.432920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.432927] LustreError: Skipped 1 previous similar message [ 7208.473880] Lustre: Skipped 5 previous similar messages [ 7211.790105] Lustre: server umount lustre-OST0000 complete [ 7221.740777] Lustre: server umount lustre-MDT0000 complete [ 7225.262308] LustreError: 222473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832842 with bad export cookie 8697171621157999996 [ 7225.276700] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7225.283501] LustreError: 222473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7225.645667] Lustre: server umount lustre-MDT0001 complete [ 7325.650125] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 7328.354869] Key type lgssc unregistered [ 7328.583911] LNet: 226180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7328.590269] LNetError: 226180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7328.608110] LNet: Removed LNI 192.168.206.114@tcp [ 7329.366148] Key type .llcrypt unregistered [ 7329.368131] Key type ._llcrypt unregistered [ 7355.817753] Key type ._llcrypt registered [ 7355.821193] Key type .llcrypt registered [ 7355.983538] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7373.099382] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7374.033777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.059710] alg: No test for adler32 (adler32-zlib) [ 7375.082416] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 7375.245515] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 7376.903171] Key type lgssc registered [ 7377.912122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7408.045171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7408.079262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.367326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7409.407814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7409.505485] Lustre: lustre-MDT0000: new disk, initializing [ 7409.586985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.603305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7413.505436] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7423.833322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.941546] Lustre: 229531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.983182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7423.986532] Lustre: Skipped 1 previous similar message [ 7424.041155] Lustre: lustre-MDT0001: new disk, initializing [ 7424.141462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7424.180098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7424.196615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7428.466275] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7437.606559] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7442.996693] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.027223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.336540] Lustre: lustre-OST0000: new disk, initializing [ 7450.342251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7450.347781] Lustre: 231497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7450.432322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7456.694644] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7458.372893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7458.386882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7458.551574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7466.827024] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7471.842744] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7471.977213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7475.528593] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7475.726722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7479.267854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7479.274256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.280441] LustreError: Skipped 1 previous similar message [ 7479.287486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7479.298411] Lustre: Skipped 1 previous similar message [ 7479.314319] Lustre: Skipped 1 previous similar message [ 7483.871743] Lustre: server umount lustre-OST0000 complete [ 7489.505817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.535062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.548065] Lustre: Skipped 1 previous similar message [ 7494.581415] Lustre: server umount lustre-MDT0000 complete [ 7497.811497] LustreError: 229522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833114 with bad export cookie 13259207779808301007 [ 7497.819214] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.820947] LustreError: 229522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7498.127463] Lustre: server umount lustre-MDT0001 complete [ 7512.640633] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7521.229138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.568543] LustreError: 234434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.581949] LustreError: 234434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7521.623643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.735697] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7526.881314] LustreError: 234435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7532.005950] LustreError: 234434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.439498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.786828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.871380] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7540.652089] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7546.337714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7546.624434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.226992] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7560.390158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.647231] Lustre: lustre-OST0001: new disk, initializing [ 7560.653854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7560.668589] Lustre: 236739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7562.405796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7562.418547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7562.491130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7566.420178] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7577.337209] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7582.378934] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7592.602643] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:33:27 (1786833207) [ 7595.489403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7595.501256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.510052] Lustre: Skipped 1 previous similar message [ 7595.512894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.539332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.563594] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.593294] Lustre: Skipped 2 previous similar messages [ 7601.172150] Lustre: server umount lustre-MDT0000 complete [ 7602.658110] LustreError: 234429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.683496] LustreError: 234429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7604.762482] LustreError: 235170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833221 with bad export cookie 13259207779808301889 [ 7604.772694] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.784265] LustreError: 235170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7605.040636] Lustre: server umount lustre-MDT0001 complete [ 7618.768677] Lustre: server umount lustre-OST0000 complete [ 7631.737786] Lustre: server umount lustre-OST0001 complete [ 7637.287456] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 7646.632873] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 7681.925267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.167709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7682.207221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7682.258344] Lustre: lustre-MDT0000: new disk, initializing [ 7682.315892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.324724] Lustre: Skipped 1 previous similar message [ 7682.345720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7686.494233] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7695.704847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.835507] Lustre: 240686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7695.933545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7695.942415] Lustre: Skipped 1 previous similar message [ 7696.078216] Lustre: lustre-MDT0001: new disk, initializing [ 7696.251675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7696.261886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7701.028829] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7710.941289] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7717.059216] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7724.228550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.512172] Lustre: lustre-OST0000: new disk, initializing [ 7724.515112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7724.520330] Lustre: 242653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7724.619155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.624028] Lustre: Skipped 1 previous similar message [ 7725.651580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7725.667844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7725.751591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7729.782754] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7738.245714] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.338072] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7742.662854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7747.613558] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7747.825504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7751.648965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.672219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.680974] Lustre: Skipped 4 previous similar messages [ 7756.133902] Lustre: server umount lustre-OST0000 complete [ 7761.889433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.897215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.903368] Lustre: Skipped 2 previous similar messages [ 7761.918617] Lustre: Skipped 1 previous similar message [ 7765.637412] Lustre: server umount lustre-MDT0000 complete [ 7767.010621] LustreError: 240696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7767.047432] LustreError: 240696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7769.277965] LustreError: 240678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833386 with bad export cookie 13259207779808304024 [ 7769.297250] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7769.726918] Lustre: server umount lustre-MDT0001 complete [ 7778.092992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.495215] LustreError: 244869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.737438] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7791.127429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.472960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7791.477896] Lustre: Skipped 1 previous similar message [ 7794.654363] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7801.653425] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7806.365430] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7812.184173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.983391] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 7824.710348] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.320874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.327417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.344929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7972.354822] Lustre: Skipped 1 previous similar message [ 7977.131362] Lustre: server umount lustre-OST0000 complete [ 7981.029124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7994.335134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7994.594991] Lustre: server umount lustre-MDT0000 complete [ 7997.925210] LustreError: 247375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.965612] LustreError: 247375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7998.460632] LustreError: 244851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833615 with bad export cookie 13259207779808304906 [ 7998.461720] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.475787] LustreError: 244851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7998.877915] Lustre: server umount lustre-MDT0001 complete [ 8090.430235] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 8093.171654] Key type lgssc unregistered [ 8093.522245] LNet: 248480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8093.548392] LNetError: 248480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8093.591967] LNet: Removed LNI 192.168.206.114@tcp [ 8094.439158] Key type .llcrypt unregistered [ 8094.441831] Key type ._llcrypt unregistered [ 8115.338817] Key type ._llcrypt registered [ 8115.341213] Key type .llcrypt registered [ 8115.439609] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8128.973323] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8129.896501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8129.939045] alg: No test for adler32 (adler32-zlib) [ 8131.054708] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8131.372264] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 8133.087493] Key type lgssc registered [ 8134.218524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8175.927939] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8184.605788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8184.634386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.880765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8185.914162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8186.053977] Lustre: lustre-MDT0000: new disk, initializing [ 8186.161941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.193343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8192.051623] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8202.230691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.394721] Lustre: 252552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8202.452801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8202.457403] Lustre: Skipped 1 previous similar message [ 8202.577805] Lustre: lustre-MDT0001: new disk, initializing [ 8202.699459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8202.737717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8202.754152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8206.015733] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8210.368234] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8216.321759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.745263] Lustre: lustre-OST0000: new disk, initializing [ 8216.756427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8216.777605] Lustre: 254185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8216.969287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.193057] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8223.293635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8223.303908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8223.400280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8231.255357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.425634] Lustre: lustre-OST0001: new disk, initializing [ 8231.430887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8231.439303] Lustre: 255054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8231.552614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8236.968875] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8238.231375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8238.243712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8238.331648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8247.565859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8255.429812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8269.290689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.294591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8269.321965] Lustre: Skipped 1 previous similar message [ 8269.323669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8269.352803] Lustre: Skipped 3 previous similar messages [ 8271.417476] Lustre: server umount lustre-MDT0000 complete [ 8274.471929] LustreError: 252543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833891 with bad export cookie 10213639047792636223 [ 8274.474470] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8274.478462] LustreError: 252543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8274.852896] Lustre: server umount lustre-MDT0001 complete [ 8288.864861] Lustre: server umount lustre-OST0000 complete [ 8302.737913] Lustre: server umount lustre-OST0001 complete [ 8319.777199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8333.027373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8340.885025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8351.422286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8368.880740] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8378.414345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.427286] Lustre: lustre-MDT0000: reset Object Index mappings [ 8378.834426] LustreError: 260064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.854382] LustreError: 260064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8378.928511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.647620] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8383.971132] LustreError: 260065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8388.070497] LustreError: 260064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.113672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.144471] Lustre: lustre-MDT0001: reset Object Index mappings [ 8390.448028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8393.895692] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8396.186399] Lustre: 261205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8401.779546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.184040] LustreError: 261557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.213332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8408.256248] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8408.551313] LustreError: 261557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.396123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.575131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8415.579221] Lustre: Skipped 1 previous similar message [ 8416.630124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8421.640948] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8428.591638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8431.427703] Lustre: 263078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8436.706044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8436.710222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.716856] Lustre: Skipped 2 previous similar messages [ 8436.720298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.380572] Lustre: server umount lustre-MDT0000 complete [ 8442.341143] LustreError: 260064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.362392] LustreError: 260064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8444.473132] LustreError: 260045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834061 with bad export cookie 10213639047792643013 [ 8444.478678] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8444.490136] LustreError: 260045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8444.867590] Lustre: server umount lustre-MDT0001 complete [ 8458.478922] Lustre: server umount lustre-OST0000 complete [ 8473.246104] Lustre: server umount lustre-OST0001 complete [ 8491.604520] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:48:26 (1786834106) [ 8508.289107] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8518.415460] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8553.374406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.676855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8553.733028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8553.849819] Lustre: lustre-MDT0000: new disk, initializing [ 8554.011163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.035310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8558.416984] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8568.010338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.162301] Lustre: 267334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8568.240818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8568.245337] Lustre: Skipped 1 previous similar message [ 8568.320619] Lustre: lustre-MDT0001: new disk, initializing [ 8568.453248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8568.466918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8572.218196] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8580.043477] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8584.429527] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.446835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.674535] Lustre: lustre-OST0000: new disk, initializing [ 8590.677090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8590.690542] Lustre: 269302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.842494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8591.851986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8591.932210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8595.473609] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8604.516384] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8609.560540] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8609.886780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8615.113263] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8615.562993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8617.440651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8617.445743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.455749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8617.475506] Lustre: Skipped 4 previous similar messages [ 8617.497852] Lustre: Skipped 5 previous similar messages [ 8622.575282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8622.587344] Lustre: Skipped 1 previous similar message [ 8623.462898] Lustre: server umount lustre-OST0000 complete [ 8627.682539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.683703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.693181] Lustre: Skipped 1 previous similar message [ 8627.705550] Lustre: Skipped 1 previous similar message [ 8632.910621] Lustre: server umount lustre-MDT0000 complete [ 8636.285442] LustreError: 267325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834253 with bad export cookie 10213639047792645099 [ 8636.291450] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8636.295742] LustreError: 267325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8636.680212] Lustre: server umount lustre-MDT0001 complete [ 8653.136789] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8662.196767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8662.706693] LustreError: 272235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.773109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8662.778746] Lustre: Skipped 2 previous similar messages [ 8667.401904] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8674.748541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.834235] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8681.164651] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8686.850577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8692.537883] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8700.007407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8700.149387] Lustre: lustre-OST0001: new disk, initializing [ 8700.153204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8700.158348] Lustre: 274542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8701.652048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8701.665893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8701.768822] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8705.437889] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8714.290980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8717.505824] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8722.072565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8722.219578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8731.616980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8731.622486] LustreError: Skipped 1 previous similar message [ 8731.634788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.654738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.665762] Lustre: Skipped 2 previous similar messages [ 8734.067082] Lustre: server umount lustre-MDT0000 complete [ 8737.523467] LustreError: 275307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834354 with bad export cookie 10213639047792645988 [ 8737.533487] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8737.543022] LustreError: 275307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8737.764184] LustreError: 275618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.777699] LustreError: 275618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8737.971571] Lustre: server umount lustre-MDT0001 complete [ 8743.537355] Lustre: server umount lustre-OST0000 complete [ 8747.059822] Lustre: server umount lustre-OST0001 complete [ 8761.145881] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8769.769391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.971096] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8782.262698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.493391] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8789.212524] Lustre: 278741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8795.499877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8795.850286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.860268] Lustre: Skipped 5 previous similar messages [ 8801.044721] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8808.110542] LustreError: 279093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.148948] LustreError: 279093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8809.029300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8814.701622] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8815.618557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8815.652217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8821.799959] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8825.166356] Lustre: 280614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8829.254878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8829.311707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8837.840175] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:54:12 (1786834452) [ 8854.496759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8854.505439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.517602] Lustre: Skipped 5 previous similar messages [ 8854.525038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.529059] Lustre: Skipped 4 previous similar messages [ 8858.371309] Lustre: server umount lustre-MDT0000 complete [ 8861.523166] LustreError: 277581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834478 with bad export cookie 10213639047792652750 [ 8861.535801] LustreError: 277581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8861.537486] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8867.879090] Lustre: server umount lustre-MDT0001 complete [ 8885.215164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8885.412709] Lustre: server umount lustre-OST0000 complete [ 8894.659585] Lustre: server umount lustre-OST0001 complete [ 8899.865512] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 8908.868221] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8949.091825] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 8958.419143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8958.677684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8958.711879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8958.792515] Lustre: lustre-MDT0000: new disk, initializing [ 8958.899081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8963.524364] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8973.074374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.181938] Lustre: 284854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8973.457447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8977.255831] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 8986.963991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.239226] Lustre: 286485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8988.646343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8992.446747] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9001.698888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9001.866656] Lustre: 287355:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9003.804653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9007.673720] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9016.498737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9019.654681] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9019.658186] Lustre: Skipped 1 previous similar message [ 9032.873331] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:57:27 (1786834647) [ 9034.744212] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.762738] Lustre: Skipped 5 previous similar messages [ 9034.769100] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9034.771687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9034.791616] Lustre: Skipped 10 previous similar messages [ 9036.980608] Lustre: server umount lustre-MDT0000 complete [ 9040.217643] LustreError: 284845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834657 with bad export cookie 10213639047792660205 [ 9040.225969] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.233183] LustreError: 284845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9040.536739] Lustre: server umount lustre-MDT0001 complete [ 9053.332548] Lustre: server umount lustre-OST0000 complete [ 9067.108954] Lustre: server umount lustre-OST0001 complete [ 9075.181612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.543762] LustreError: 289603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.555132] LustreError: 289603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9075.646394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.650767] Lustre: Skipped 5 previous similar messages [ 9079.835334] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9087.240721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.998327] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9098.630469] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9103.947798] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9110.162868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.899418] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9121.951830] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9135.358109] loop1: detected capacity change from 0 to 204800000 [ 9135.525597] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9135.540739] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9135.557942] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9139.819829] loop1: detected capacity change from 0 to 204800000 [ 9145.013777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.090215] LustreError: 289593:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10394: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9145.098178] LustreError: 289593:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10394 [ 9145.101864] LustreError: 289593:0:(mgs_handler.c:517:mgs_target_reg()) fs10394-MDT0001: Error checking target index: rc = -13 [ 9145.107990] LustreError: fs10394-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9145.113911] LustreError: 293322:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9145.118463] LustreError: 293322:0:(tgt_mount.c:2129:server_put_super()) no obd fs10394-MDT0001 [ 9145.126963] LustreError: 293322:0:(tgt_mount.c:116:server_deregister_mount()) fs10394-MDT0001 not registered [ 9145.159605] Lustre: server umount fs10394-MDT0001 complete [ 9145.161677] LustreError: 293322:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9152.503879] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.579547] Lustre: Setting parameter fs10394-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10394-MDT0001 [ 9152.627060] Lustre: srv-fs10394-MDT0001: No data found on store. Initialize space. [ 9152.630576] Lustre: Skipped 4 previous similar messages [ 9152.681271] Lustre: fs10394-MDT0001: new disk, initializing [ 9152.683710] Lustre: Skipped 3 previous similar messages [ 9152.775346] Lustre: srv-fs10394-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9152.784201] Lustre: 293788:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10394-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9152.790459] LustreError: 293788:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10394-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9156.409413] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9161.951691] Lustre: server umount fs10394-MDT0001 complete [ 9172.459143] loop1: detected capacity change from 0 to 204800000 [ 9176.798398] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.077758] Lustre: 295352:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10394-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9177.097147] LustreError: 295352:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10394-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9180.241693] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9184.126750] Lustre: server umount fs10394-MDT0001 complete [ 9192.426116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.429346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.438082] Lustre: Skipped 5 previous similar messages [ 9192.444785] Lustre: Skipped 5 previous similar messages [ 9197.547518] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9197.859158] Lustre: server umount lustre-MDT0000 complete [ 9204.516935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.608841] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9208.346926] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9209.846596] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9214.608968] loop1: detected capacity change from 0 to 204800000 [ 9218.560553] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9218.846521] Lustre: 297798:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10394-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9218.852562] LustreError: 297798:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10394-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9222.507832] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9226.375533] Lustre: server umount fs10394-MDT0001 complete [ 9245.681543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9249.576797] Lustre: server umount lustre-OST0000 complete [ 9258.778289] Lustre: server umount lustre-MDT0000 complete [ 9261.517286] LustreError: 289585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834878 with bad export cookie 10213639047792662886 [ 9261.518078] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9261.526890] LustreError: 289585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9261.777340] Lustre: server umount lustre-MDT0001 complete [ 9269.301573] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 9271.321190] Key type lgssc unregistered [ 9271.570931] LNet: 300152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9271.579297] LNetError: 300152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9272.613667] LNet: Removed LNI 192.168.206.114@tcp [ 9273.388164] Key type .llcrypt unregistered [ 9273.390275] Key type ._llcrypt unregistered [ 9287.884534] Key type ._llcrypt registered [ 9287.886701] Key type .llcrypt registered [ 9287.954723] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9288.626985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9288.658917] alg: No test for adler32 (adler32-zlib) [ 9289.611233] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9289.775822] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 9291.384042] Key type lgssc registered [ 9292.232353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9298.412145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9298.428408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.881454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.428667] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9309.162549] LustreError: 301404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.184747] LustreError: 301404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9310.105452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.512309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9313.899193] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9319.856948] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9324.519807] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9329.460963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9329.716521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9334.554546] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9335.848371] LustreError: 303232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.138753] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9340.897919] LustreError: 303233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.020167] LustreError: 303232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.039692] LustreError: 303232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9352.047682] LustreError: 303686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9352.065569] LustreError: 303686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9358.353991] loop1: detected capacity change from 0 to 204800000 [ 9358.529915] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9358.538984] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9358.546864] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9362.211253] loop1: detected capacity change from 0 to 204800000 [ 9362.301041] LustreError: 303233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.319722] LustreError: 303233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9366.215841] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.329123] Lustre: fs22276-OST0000: new disk, initializing [ 9366.331654] Lustre: srv-fs22276-OST0000: No data found on store. Initialize space. [ 9366.335822] Lustre: 305138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9366.397869] Lustre: fs22276-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.363926] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9376.123735] Lustre: server umount fs22276-OST0000 complete [ 9378.790088] LustreError: 303686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9378.799724] LustreError: 303686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 9384.158419] loop1: detected capacity change from 0 to 204800000 [ 9387.911027] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9388.036188] Lustre: fs22276-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.722500] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9397.386543] loop2: detected capacity change from 0 to 204800000 [ 9397.530053] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9397.538439] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9397.547494] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9400.929871] loop2: detected capacity change from 0 to 204800000 [ 9404.552427] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9404.602943] LustreError: fs22276-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9404.613208] LustreError: fs22276-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9404.620982] LustreError: 308178:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9404.627938] LustreError: 308178:0:(tgt_mount.c:2129:server_put_super()) no obd fs22276-OST0001 [ 9404.632742] LustreError: 308178:0:(tgt_mount.c:116:server_deregister_mount()) fs22276-OST0001 not registered [ 9404.663489] Lustre: server umount fs22276-OST0001 complete [ 9404.667222] LustreError: 308178:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9411.123737] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9411.212321] Lustre: fs22276-OST0001: new disk, initializing [ 9411.216975] Lustre: srv-fs22276-OST0001: No data found on store. Initialize space. [ 9411.221679] Lustre: 308730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9411.278900] Lustre: fs22276-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.485438] LustreError: 303232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9413.504201] LustreError: 303232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 9415.120817] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9419.751292] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9420.547696] Lustre: server umount fs22276-OST0000 complete [ 9424.874133] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9426.584306] Lustre: server umount fs22276-OST0001 complete [ 9434.958839] loop1: detected capacity change from 0 to 204800000 [ 9438.771211] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9438.953784] Lustre: fs22276-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9443.207387] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9447.751823] loop2: detected capacity change from 0 to 204800000 [ 9451.240574] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.155893] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9460.809738] Lustre: server umount fs22276-OST0000 complete [ 9466.893988] Lustre: server umount fs22276-OST0001 complete [ 9476.064184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.070322] Lustre: Skipped 1 previous similar message [ 9476.073819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9476.078400] Lustre: Skipped 1 previous similar message [ 9480.658647] Lustre: server umount lustre-OST0000 complete [ 9484.769033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9484.778944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9484.795161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.304834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.305439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.322783] Lustre: Skipped 1 previous similar message [ 9491.431366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.434796] Lustre: Skipped 1 previous similar message [ 9496.545634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9496.550855] Lustre: Skipped 1 previous similar message [ 9498.079132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9498.183497] Lustre: server umount lustre-MDT0000 complete [ 9500.817767] LustreError: 301385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835117 with bad export cookie 6029714876848219137 [ 9500.820331] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9500.824318] LustreError: 301385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9501.050958] Lustre: server umount lustre-MDT0001 complete [ 9508.749387] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 9510.845647] Key type lgssc unregistered [ 9511.065688] LNet: 314651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9511.080290] LNetError: 314651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9511.096749] LNet: Removed LNI 192.168.206.114@tcp [ 9511.805423] Key type .llcrypt unregistered [ 9511.809257] Key type ._llcrypt unregistered [ 9532.763522] Key type ._llcrypt registered [ 9532.765665] Key type .llcrypt registered [ 9532.838709] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 9543.674569] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9544.318624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9544.376783] alg: No test for adler32 (adler32-zlib) [ 9545.332680] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9545.499596] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 9547.135211] Key type lgssc registered [ 9547.943340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9571.256352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9571.273995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.446724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9572.470108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9572.526254] Lustre: lustre-MDT0000: new disk, initializing [ 9572.595813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9572.613981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9575.720085] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9582.843299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.954708] Lustre: 318483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9583.000782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9583.007116] Lustre: Skipped 1 previous similar message [ 9583.073309] Lustre: lustre-MDT0001: new disk, initializing [ 9583.125109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9583.155822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9583.163308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9586.025773] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9593.142188] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9597.083720] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9601.803151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9601.996411] Lustre: lustre-OST0000: new disk, initializing [ 9601.998967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9602.003800] Lustre: 320450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9602.077272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9606.430804] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9608.219166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9608.227170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9608.290196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9613.170653] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9614.678259] Lustre: Mounted lustre-client - version 2.17.57_5_g4421514 [ 9615.364870] Lustre: Unmounted lustre-client [ 9623.521418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9623.526554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.528300] LustreError: Skipped 1 previous similar message [ 9623.530729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9623.560465] Lustre: Skipped 1 previous similar message [ 9625.064544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9625.071170] Lustre: Skipped 2 previous similar messages [ 9630.182810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9630.187419] Lustre: Skipped 1 previous similar message [ 9635.303992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9635.310674] Lustre: Skipped 1 previous similar message [ 9636.319391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9636.427394] Lustre: server umount lustre-OST0000 complete [ 9639.394530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9639.403957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.415454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9640.416801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.657355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9650.664959] Lustre: Skipped 5 previous similar messages [ 9653.215450] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9653.361585] Lustre: server umount lustre-MDT0000 complete [ 9655.938790] LustreError: 318475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835272 with bad export cookie 1920672529172631869 [ 9655.946716] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9655.948128] LustreError: 318475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9656.160204] Lustre: server umount lustre-MDT0001 complete [ 9663.365404] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 9665.344426] Key type lgssc unregistered [ 9675.551823] LNet: 1 local NIs in recovery (showing 1): 192.168.206.114@tcp [ 9675.808509] LNet: 322617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9675.815777] LNetError: 322617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9745.446315] LNet: Removed LNI 192.168.206.114@tcp [ 9745.981932] Key type .llcrypt unregistered [ 9745.985119] Key type ._llcrypt unregistered [ 9758.416261] Key type ._llcrypt registered [ 9758.419320] Key type .llcrypt registered [ 9758.515578] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [ 9768.772736] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9769.434187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9769.516773] alg: No test for adler32 (adler32-zlib) [ 9770.504789] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9770.656548] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 9772.263244] Key type lgssc registered [ 9772.986697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9793.467359] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:10:08 (1786835408) [ 9797.510572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9797.528921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.679904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9798.702675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9798.761245] Lustre: lustre-MDT0000: new disk, initializing [ 9798.819607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.833370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9802.015365] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9809.792392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.892500] Lustre: 325817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9809.947503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9809.950742] Lustre: Skipped 1 previous similar message [ 9810.004323] Lustre: lustre-MDT0001: new disk, initializing [ 9810.073098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9810.104220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9810.110147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9813.366487] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9819.726770] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9823.094345] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9827.227091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9827.445770] Lustre: lustre-OST0000: new disk, initializing [ 9827.448953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9827.454260] Lustre: 327783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9827.517384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.482198] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9834.527264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9834.535766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9834.634680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9837.648140] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9849.825650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9849.828307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.831409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9849.841325] Lustre: Skipped 1 previous similar message [ 9852.177771] Lustre: server umount lustre-OST0000 complete [ 9854.948409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.949182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9854.964523] Lustre: Skipped 1 previous similar message [ 9854.968222] Lustre: Skipped 1 previous similar message [ 9860.064509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9860.068602] Lustre: Skipped 2 previous similar messages [ 9860.615940] Lustre: server umount lustre-MDT0000 complete [ 9862.988438] LustreError: 325808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835479 with bad export cookie 15685388012840331560 [ 9862.990157] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9863.001255] LustreError: 325808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9863.211987] Lustre: server umount lustre-MDT0001 complete [ 9869.583393] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [ 9871.341905] Key type lgssc unregistered [ 9871.554981] LNet: 329688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9871.564386] LNetError: 329688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9871.579513] LNet: Removed LNI 192.168.206.114@tcp [ 9872.104159] Key type .llcrypt unregistered [ 9872.105927] Key type ._llcrypt unregistered [ 9884.973685] Key type ._llcrypt registered [ 9884.975177] Key type .llcrypt registered [ 9885.043442] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [ 9885.571076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9885.615280] alg: No test for adler32 (adler32-zlib) [ 9886.566658] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9886.750406] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [ 9888.367939] Key type lgssc registered [ 9889.010300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9893.351393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9893.367019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9894.630942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9897.106692] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9901.687572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.882385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.219255] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9908.382978] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9911.292286] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9914.601784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9914.844174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.104404] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9922.110873] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9932.417244] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9933.271725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9936.352685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.357149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.360981] Lustre: Skipped 1 previous similar message [ 9939.838040] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.14@tcp (stopping) [ 9939.842136] Lustre: Skipped 2 previous similar messages [ 9941.474418] LustreError: 330940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9941.494803] LustreError: 330940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9941.537255] Lustre: server umount lustre-MDT0000 complete [ 9943.440923] LustreError: 330919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835560 with bad export cookie 14752107093463313268 [ 9943.446167] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9943.447386] LustreError: 330919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9944.940584] LustreError: 330935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.961274] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.14@tcp (stopping) [ 9946.593000] LustreError: 331995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.594744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9946.608844] Lustre: Skipped 1 previous similar message [ 9948.640019] LustreError: 330936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.641090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9948.652028] Lustre: Skipped 1 previous similar message [ 9949.658355] Lustre: server umount lustre-MDT0001 complete [ 9957.437158] Lustre: server umount lustre-OST0000 complete [ 9961.877596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.866573] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9965.179146] Lustre: 335115:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.114@tcp [ 9966.210232] Lustre: server umount MGS complete [ 9970.539862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.775985] LustreError: 335686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.825740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.846329] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9977.249962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9977.439700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9979.401547] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9983.455605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.602870] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [ 9990.202587] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9990.988407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9993.784458] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10001.715268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [10006.499438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.500042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10006.501339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10006.506958] Lustre: Skipped 1 previous similar message [10009.832166] Lustre: server umount lustre-OST0000 complete [10011.616377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.623113] Lustre: Skipped 1 previous similar message [10016.736883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.739684] Lustre: Skipped 4 previous similar messages [10017.776664] Lustre: server umount lustre-MDT0000 complete [10019.615237] LustreError: 337143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835636 with bad export cookie 14752107093463314108 [10019.618903] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10019.624842] LustreError: 337143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10019.809501] Lustre: server umount lustre-MDT0001 complete [10025.615154] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10027.564931] Key type lgssc unregistered [10027.751918] LNet: 339120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10027.758733] LNetError: 339120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10027.772167] LNet: Removed LNI 192.168.206.114@tcp [10028.359838] Key type .llcrypt unregistered [10028.362919] Key type ._llcrypt unregistered [10039.324686] Key type ._llcrypt registered [10039.329447] Key type .llcrypt registered [10039.395756] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10048.899172] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10049.371244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10049.452212] alg: No test for adler32 (adler32-zlib) [10050.397693] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10050.531638] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [10052.159184] Key type lgssc registered [10052.716041] Lustre: Echo OBD driver; http://www.lustre.org/ [10072.725097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10072.738849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10073.869923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10073.893228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10073.941372] Lustre: lustre-MDT0000: new disk, initializing [10073.988404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10074.003987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10076.699772] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10083.349873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10083.425466] Lustre: 342260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10083.471623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10083.476092] Lustre: Skipped 1 previous similar message [10083.530252] Lustre: lustre-MDT0001: new disk, initializing [10083.587974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10083.616170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10083.622694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10086.080269] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10091.640346] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10095.042849] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10099.078916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10099.249571] Lustre: lustre-OST0000: new disk, initializing [10099.252625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10099.258454] Lustre: 344228:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10099.318960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10102.796702] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10107.425707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10107.431702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10107.513398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10108.692391] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10111.317432] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10111.437815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10113.880855] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10114.010716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10117.602430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10117.604618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10117.609680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10117.617880] Lustre: Skipped 1 previous similar message [10117.626887] Lustre: Skipped 1 previous similar message [10122.722889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10122.727403] Lustre: Skipped 1 previous similar message [10127.843365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10127.849129] Lustre: Skipped 1 previous similar message [10129.375345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10129.464296] Lustre: server umount lustre-OST0000 complete [10132.962950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.966503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10132.971114] Lustre: Skipped 1 previous similar message [10138.081543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10138.088636] Lustre: Skipped 2 previous similar messages [10146.271223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10146.388420] Lustre: server umount lustre-MDT0000 complete [10148.665723] LustreError: 342253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835765 with bad export cookie 6283495059191557369 [10148.667662] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10148.674500] LustreError: 342253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10148.939731] Lustre: server umount lustre-MDT0001 complete [10154.257594] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:16:09 (1786835769) [10162.141720] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10166.966965] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10185.686454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.852392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10185.876754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10185.927955] Lustre: lustre-MDT0000: new disk, initializing [10185.975269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10185.995313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10188.313557] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10194.631921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.686721] Lustre: 348979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10194.719272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10194.722853] Lustre: Skipped 1 previous similar message [10194.767670] Lustre: lustre-MDT0001: new disk, initializing [10194.826452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10194.859500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10194.866814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10196.954021] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10202.320942] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.313677] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10209.014676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10209.169367] Lustre: lustre-OST0000: new disk, initializing [10209.172126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10209.178059] Lustre: 350948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10209.233687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10212.197227] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10216.475807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10216.490142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10216.547466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10217.598808] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10236.899518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10236.900472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.910676] LustreError: Skipped 1 previous similar message [10236.920436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.925912] Lustre: Skipped 3 previous similar messages [10239.058284] Lustre: server umount lustre-OST0000 complete [10242.017577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.028099] Lustre: Skipped 1 previous similar message [10247.375400] Lustre: server umount lustre-MDT0000 complete [10249.412242] LustreError: 349017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835866 with bad export cookie 6283495059191558230 [10249.414961] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10249.417704] LustreError: 349017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10249.629104] Lustre: server umount lustre-MDT0001 complete [10255.036272] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10256.618591] Key type lgssc unregistered [10256.785987] LNet: 353101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10256.791745] LNetError: 353101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10257.830335] LNet: Removed LNI 192.168.206.114@tcp [10258.259062] Key type .llcrypt unregistered [10258.261264] Key type ._llcrypt unregistered [10307.592977] Key type ._llcrypt registered [10307.595013] Key type .llcrypt registered [10307.660752] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10307.833419] Key type .llcrypt unregistered [10307.835355] Key type ._llcrypt unregistered [10347.917581] Key type ._llcrypt registered [10347.918776] Key type .llcrypt registered [10347.962779] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10348.113323] Key type .llcrypt unregistered [10348.114439] Key type ._llcrypt unregistered [10361.798476] Key type ._llcrypt registered [10361.800342] Key type .llcrypt registered [10361.849660] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10362.017490] Key type .llcrypt unregistered [10362.022483] Key type ._llcrypt unregistered [10376.879920] Key type ._llcrypt registered [10376.881961] Key type .llcrypt registered [10376.944577] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10377.111279] Key type .llcrypt unregistered [10377.113230] Key type ._llcrypt unregistered [10415.028545] Key type ._llcrypt registered [10415.031753] Key type .llcrypt registered [10415.098224] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10415.288692] Key type .llcrypt unregistered [10415.291334] Key type ._llcrypt unregistered [10430.654773] Key type ._llcrypt registered [10430.657457] Key type .llcrypt registered [10430.705491] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10430.869992] Key type .llcrypt unregistered [10430.874944] Key type ._llcrypt unregistered [10445.860535] Key type ._llcrypt registered [10445.862442] Key type .llcrypt registered [10445.916967] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10446.124442] Key type .llcrypt unregistered [10446.127389] Key type ._llcrypt unregistered [10460.736528] Key type ._llcrypt registered [10460.738792] Key type .llcrypt registered [10460.798409] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10460.953278] Key type .llcrypt unregistered [10460.954927] Key type ._llcrypt unregistered [10475.977045] Key type ._llcrypt registered [10475.978843] Key type .llcrypt registered [10476.030966] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10476.186558] Key type .llcrypt unregistered [10476.188239] Key type ._llcrypt unregistered [10490.705878] Key type ._llcrypt registered [10490.708610] Key type .llcrypt registered [10490.776426] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10490.947366] Key type .llcrypt unregistered [10490.951870] Key type ._llcrypt unregistered [10521.214423] Key type ._llcrypt registered [10521.216758] Key type .llcrypt registered [10521.279695] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10521.461329] Key type .llcrypt unregistered [10521.465783] Key type ._llcrypt unregistered [10536.819863] Key type ._llcrypt registered [10536.822720] Key type .llcrypt registered [10536.887479] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10537.040321] Key type .llcrypt unregistered [10537.042476] Key type ._llcrypt unregistered [10553.066359] Key type ._llcrypt registered [10553.067621] Key type .llcrypt registered [10553.122312] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10553.269972] Key type .llcrypt unregistered [10553.274088] Key type ._llcrypt unregistered [10577.609521] Key type ._llcrypt registered [10577.611585] Key type .llcrypt registered [10577.661136] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10585.611523] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10586.085734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10586.094270] alg: No test for adler32 (adler32-zlib) [10587.002882] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10587.117409] LNet: Added LNI 192.168.206.114@tcp [8/256/0/180] [10588.711240] Key type lgssc registered [10589.168039] Lustre: Echo OBD driver; http://www.lustre.org/ [10603.779215] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10603.796487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10604.893290] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10604.910203] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10604.950127] Lustre: tst250fs-MDT0000: new disk, initializing [10604.982464] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10604.995036] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10606.748272] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10612.097670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.157986] Lustre: 373070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10612.188252] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10612.191464] Lustre: Skipped 1 previous similar message [10612.231884] Lustre: tst250fs-MDT0001: new disk, initializing [10612.277503] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10612.296256] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10612.301576] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10614.167619] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10616.988057] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10620.328562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.481306] Lustre: tst250fs-OST0000: new disk, initializing [10620.483603] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10620.487120] Lustre: 374702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10620.536621] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10623.357462] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10627.082847] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10627.089231] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10627.149292] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10628.581435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10628.662559] Lustre: tst250fs-OST0001: new disk, initializing [10628.666240] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10628.670113] Lustre: 375573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10628.721560] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10631.334688] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_default_debug -1 all [10634.574454] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10637.053071] Lustre: 373072:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.114@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [10648.032104] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10648.035844] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10648.043149] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10649.569477] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10649.574439] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10649.580962] Lustre: Skipped 1 previous similar message [10652.833121] Lustre: server umount tst250fs-MDT0000 complete [10654.330841] LustreError: 373062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836271 with bad export cookie 2554930390878658410 [10654.335180] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.335989] LustreError: 373062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10654.470977] Lustre: server umount tst250fs-MDT0001 complete [10665.101065] Lustre: server umount tst250fs-OST0000 complete [10676.839752] Lustre: server umount tst250fs-OST0001 complete [10684.573222] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10687.915560] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10700.774583] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:25:16 (1786836316) [10701.436485] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10708.044942] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing set_hostid [10712.255605] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing load_modules_local [10724.724537] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10459 sec ========= 19:25:40 (1786836340) [10725.389609] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:25:40 (1786836340) === [10726.777625] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:25:42 (1786836342) === [10741.941066] Lustre: DEBUG MARKER: oleg614-server.virtnet: executing unload_modules_local [10743.137639] Key type lgssc unregistered [10743.267507] LNet: 384681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10743.270394] LNetError: 384681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10743.278469] LNet: Removed LNI 192.168.206.114@tcp [10743.652209] Key type .llcrypt unregistered [10743.653920] Key type ._llcrypt unregistered