[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470491458 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2624MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013042] *** VALIDATE selinux *** [ 0.021783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026115] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028026] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] *** VALIDATE tmpfs *** [ 0.032472] *** VALIDATE proc *** [ 0.033256] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040025] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043819] debug: unmapping init [mem 0xffffffff91c59000-0xffffffff91c60fff] [ 0.045974] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046832] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049017] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054387] rcu: Hierarchical SRCU implementation. [ 0.056677] smp: Bringing up secondary CPUs ... [ 0.057661] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.062205] smp: Brought up 1 node, 4 CPUs [ 0.064088] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138718] node 0 deferred pages initialised in 71ms [ 0.142062] devtmpfs: initialized [ 0.143229] x86/mm: Memory block size: 128MB [ 0.145371] gcov: version magic: 0x41383552 [ 0.148300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149209] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150355] pinctrl core: initialized pinctrl subsystem [ 0.151214] [ 0.151787] ************************************************************* [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153015] ** ** [ 0.154014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155013] ** ** [ 0.156016] ** This means that this kernel is built to expose internal ** [ 0.157019] ** IOMMU data structures, which may compromise security on ** [ 0.158015] ** your system. ** [ 0.159014] ** ** [ 0.160013] ** If you see this message and you are not debugging the ** [ 0.161013] ** kernel, report this immediately to your vendor! ** [ 0.162014] ** ** [ 0.163014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164015] ************************************************************* [ 0.165768] NET: Registered protocol family 16 [ 0.166513] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167124] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170065] cpuidle: using governor menu [ 0.172217] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175466] PCI: Using configuration type 1 for base access [ 0.178127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188167] cryptd: max_cpu_qlen set to 1000 [ 0.190261] ACPI: Added _OSI(Module Device) [ 0.191012] ACPI: Added _OSI(Processor Device) [ 0.192014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197094] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199697] ACPI: Interpreter enabled [ 0.200056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201009] ACPI: Using IOAPIC for interrupt routing [ 0.202107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.203395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213204] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218389] acpiphp: Slot [2] registered [ 0.219102] acpiphp: Slot [5] registered [ 0.220082] acpiphp: Slot [6] registered [ 0.221089] acpiphp: Slot [7] registered [ 0.222073] acpiphp: Slot [8] registered [ 0.223100] acpiphp: Slot [9] registered [ 0.224131] acpiphp: Slot [10] registered [ 0.225143] acpiphp: Slot [3] registered [ 0.226089] acpiphp: Slot [4] registered [ 0.227100] acpiphp: Slot [11] registered [ 0.228103] acpiphp: Slot [12] registered [ 0.229111] acpiphp: Slot [13] registered [ 0.230133] acpiphp: Slot [14] registered [ 0.231113] acpiphp: Slot [15] registered [ 0.232108] acpiphp: Slot [16] registered [ 0.233094] acpiphp: Slot [17] registered [ 0.234105] acpiphp: Slot [18] registered [ 0.235112] acpiphp: Slot [19] registered [ 0.236100] acpiphp: Slot [20] registered [ 0.237135] acpiphp: Slot [21] registered [ 0.238104] acpiphp: Slot [22] registered [ 0.239104] acpiphp: Slot [23] registered [ 0.240104] acpiphp: Slot [24] registered [ 0.241123] acpiphp: Slot [25] registered [ 0.242130] acpiphp: Slot [26] registered [ 0.243126] acpiphp: Slot [27] registered [ 0.244192] acpiphp: Slot [28] registered [ 0.245176] acpiphp: Slot [29] registered [ 0.247111] acpiphp: Slot [30] registered [ 0.248126] acpiphp: Slot [31] registered [ 0.250206] PCI host bridge to bus 0000:00 [ 0.252026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.255026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.257025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.263026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.267228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270816] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.273210] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.282012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.287185] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.290021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.292020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.298132] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.305048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.308930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.313016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.324028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.328017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332195] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.350015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.374020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395220] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.414016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468378] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.483030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.516019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530670] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.569522] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.576019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.604018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612576] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.620015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.627017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.649495] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.652366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.654317] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.656397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.659733] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.665032] iommu: Default domain type: Passthrough [ 0.666414] SCSI subsystem initialized [ 0.668132] ACPI: bus type USB registered [ 0.669081] usbcore: registered new interface driver usbfs [ 0.671100] usbcore: registered new interface driver hub [ 0.672086] usbcore: registered new device driver usb [ 0.674130] pps_core: LinuxPPS API ver. 1 registered [ 0.675018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678087] PTP clock support registered [ 0.680136] EDAC MC: Ver: 3.0.0 [ 0.681520] PCI: Using ACPI for IRQ routing [ 0.682605] NetLabel: Initializing [ 0.684026] NetLabel: domain hash size = 128 [ 0.686014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.689086] NetLabel: unlabeled traffic allowed by default [ 0.691131] vgaarb: loaded [ 0.693270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.695015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.700551] clocksource: Switched to clocksource kvm-clock [ 0.786916] VFS: Disk quotas dquot_6.6.0 [ 0.788697] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.791268] *** VALIDATE ramfs *** [ 0.792570] *** VALIDATE hugetlbfs *** [ 0.794584] pnp: PnP ACPI init [ 0.796930] pnp: PnP ACPI: found 6 devices [ 0.813052] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816316] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.818712] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.820840] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.823150] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.825395] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.827899] NET: Registered protocol family 2 [ 0.830111] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834457] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.837906] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.842730] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.846427] TCP: Hash tables configured (established 65536 bind 65536) [ 0.848812] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.851650] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854148] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857146] NET: Registered protocol family 1 [ 0.859714] RPC: Registered named UNIX socket transport module. [ 0.861798] RPC: Registered udp transport module. [ 0.863648] RPC: Registered tcp transport module. [ 0.865164] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.867262] NET: Registered protocol family 44 [ 0.868751] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.870827] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.873266] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.875344] PCI: CLS 0 bytes, default 64 [ 0.877273] Unpacking initramfs... [ 2.205211] debug: unmapping init [mem 0xffff95cbfcc54000-0xffff95cbfffbffff] [ 2.209361] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.211909] software IO TLB: mapped [mem 0x00000000a0000000-0x00000000a4000000] (64MB) [ 2.214961] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.673597] Initialise system trusted keyrings [ 2.675457] Key type blacklist registered [ 2.676959] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.683386] zbud: loaded [ 2.685585] *** VALIDATE nfs *** [ 2.686467] *** VALIDATE nfs4 *** [ 2.687444] pstore: using deflate compression [ 2.690084] Platform Keyring initialized [ 2.785785] NET: Registered protocol family 38 [ 2.787494] Key type asymmetric registered [ 2.789268] Asymmetric key parser 'x509' registered [ 2.790522] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.792608] io scheduler mq-deadline registered [ 2.793855] io scheduler kyber registered [ 2.795213] io scheduler bfq registered [ 2.796552] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.800202] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.803662] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.806846] ACPI: Power Button [PWRF] [ 2.813740] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.822325] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.840750] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.849371] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.864244] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.892645] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.920717] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.925040] Non-volatile memory driver v1.3 [ 2.926470] Linux agpgart interface v0.103 [ 2.960586] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.963970] vda: detected capacity change from 0 to 75108352 [ 2.981393] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.985725] vdb: detected capacity change from 0 to 1073741824 [ 3.002898] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.006814] vdc: detected capacity change from 0 to 2621440000 [ 3.022360] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.026615] vdd: detected capacity change from 0 to 2621440000 [ 3.047623] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.050684] vde: detected capacity change from 0 to 4294967296 [ 3.072676] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.076066] vdf: detected capacity change from 0 to 4294967296 [ 3.085454] libphy: Fixed MDIO Bus: probed [ 3.100557] usbcore: registered new interface driver usbserial_generic [ 3.104433] usbserial: USB Serial support registered for generic [ 3.107250] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.112460] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.114518] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.116976] mousedev: PS/2 mouse device common for all mice [ 3.119528] rtc_cmos 00:05: RTC can wake from S4 [ 3.123045] rtc_cmos 00:05: registered as rtc0 [ 3.124540] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.124784] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.129412] intel_pstate: CPU model not supported [ 3.132228] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.133365] hid: raw HID events driver (C) Jiri Kosina [ 3.137150] usbcore: registered new interface driver usbhid [ 3.139024] usbhid: USB HID core driver [ 3.139212] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.140143] drop_monitor: Initializing network drop monitor service [ 3.144627] Initializing XFRM netlink socket [ 3.146400] NET: Registered protocol family 10 [ 3.148652] Segment Routing with IPv6 [ 3.149888] NET: Registered protocol family 17 [ 3.151431] mpls_gso: MPLS GSO support [ 3.157965] RAS: Correctable Errors collector initialized. [ 3.160075] AVX version of gcm_enc/dec engaged. [ 3.161763] AES CTR mode by8 optimization enabled [ 3.227400] sched_clock: Marking stable (3227379567, 0)->(4203713915, -976334348) [ 3.230258] registered taskstats version 1 [ 3.231710] Loading compiled-in X.509 certificates [ 3.233119] zswap: loaded using pool lzo/zbud [ 3.257121] Key type big_key registered [ 3.268307] Key type encrypted registered [ 3.270098] ima: No TPM chip found, activating TPM-bypass! [ 3.272288] ima: Allocated hash algorithm: sha1 [ 3.274151] ima: No architecture policies found [ 3.276167] evm: Initialising EVM extended attributes: [ 3.278205] evm: security.selinux [ 3.279492] evm: security.ima [ 3.280612] evm: security.capability [ 3.282017] evm: HMAC attrs: 0x1 [ 3.284441] rtc_cmos 00:05: setting system clock to 2026-09-08 01:38:41 UTC (1788831521) [ 3.290810] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.293844] debug: unmapping init [mem 0xffffffff91982000-0xffffffff91c58fff] [ 3.300072] Write protecting the kernel read-only data: 28672k [ 3.303422] debug: unmapping init [mem 0xffffffff90003000-0xffffffff901fffff] [ 3.306470] debug: unmapping init [mem 0xffffffff90914000-0xffffffff909fffff] [ 3.337525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.346406] systemd[1]: Detected virtualization kvm. [ 3.348361] systemd[1]: Detected architecture x86-64. [ 3.350343] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.377953] systemd[1]: No hostname configured. [ 3.379799] systemd[1]: Set hostname to . [ 3.381927] random: systemd: uninitialized urandom read (16 bytes read) [ 3.384499] systemd[1]: Initializing machine ID from random generator. [ 3.433308] random: ln: uninitialized urandom read (6 bytes read) [ 3.498694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.501869] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.506102] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.510752] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.037550] device-mapper: uevent: version 1.0.3 [ 4.040353] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.657307] random: fast init done [ 4.673747] virtio_net virtio0 ens2: renamed from eth0 [ 4.701829] scsi host0: ata_piix [ 4.730308] scsi host1: ata_piix [ 4.731879] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.734407] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.350952] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.789333] random: crng init done [ 9.792380] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.120810] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.293848] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.529429] SELinux: Disabled at runtime. [ 11.590512] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.599926] systemd[1]: Detected virtualization kvm. [ 11.601795] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.115437] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.117714] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.121421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.124212] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.126329] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.131925] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.134812] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ 12.269675] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.594660] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.859335] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.950050] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.113584] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.136331] EDAC sbridge: Ver: 1.1.2 [ 14.852094] Key type dns_resolver registered [ 15.150053] NFS: Registering the id_resolver key type [ 15.152059] Key type id_resolver registered [ 15.153632] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg127-server login: [ 47.539573] hrtimer: interrupt took 4563024 ns [ 49.223559] libcfs: loading out-of-tree module taints kernel. [ 49.293061] Key type ._llcrypt registered [ 49.297187] Key type .llcrypt registered [ 49.472477] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 67.662975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 69.518397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.537455] alg: No test for adler32 (adler32-zlib) [ 71.163597] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 72.353979] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 74.175882] Key type lgssc registered [ 76.118431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.747477] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.106692] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 167.723858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.767513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.189922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.221531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.356505] Lustre: lustre-MDT0000: new disk, initializing [ 169.544825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.581782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.384400] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.074216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.179238] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.225867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.230124] Lustre: Skipped 1 previous similar message [ 187.322874] Lustre: lustre-MDT0001: new disk, initializing [ 187.392400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.433603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.443131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.576402] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.782473] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.092880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.342963] Lustre: lustre-OST0000: new disk, initializing [ 206.351304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.359216] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.453220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.524321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.536288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.710489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.509589] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.614701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.724740] Lustre: lustre-OST0001: new disk, initializing [ 226.727480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.730848] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.781114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.794455] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.571406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.576797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.674705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.469690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.726487] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.118823] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.978897] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing yml_node [ 267.421188] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 269.805548] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 272.517935] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 274.272529] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 21:43:11 EDT 2026 [ 293.316653] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 295.346482] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.801834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.811419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.816818] Lustre: Skipped 2 previous similar messages [ 312.829806] Lustre: Skipped 3 previous similar messages [ 317.931598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.943018] Lustre: Skipped 3 previous similar messages [ 318.645470] Lustre: server umount lustre-MDT0000 complete [ 322.415649] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831840 with bad export cookie 17241503651842490759 [ 322.419612] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.422381] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 323.044184] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.049725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.069322] Lustre: Skipped 2 previous similar messages [ 323.080877] Lustre: Skipped 1 previous similar message [ 327.073477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.088043] Lustre: Skipped 1 previous similar message [ 329.040644] Lustre: server umount lustre-MDT0001 complete [ 334.345627] Lustre: server umount lustre-OST0000 complete [ 339.036333] Lustre: server umount lustre-OST0001 complete [ 345.993479] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 355.865379] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 396.606399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.826409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.868849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.971183] Lustre: lustre-MDT0000: new disk, initializing [ 397.126754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.155234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.854902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 412.642303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.825337] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.871036] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.874685] Lustre: Skipped 1 previous similar message [ 412.969639] Lustre: lustre-MDT0001: new disk, initializing [ 413.081269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.137495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.151629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.802649] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 426.711127] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.356432] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.977145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.398217] Lustre: lustre-OST0000: new disk, initializing [ 442.401534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.410621] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.506135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.347222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.357224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.525454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 450.619311] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 460.358842] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.801821] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 466.092190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.954843] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.180541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.107374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.115107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.118668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.133774] LustreError: Skipped 1 previous similar message [ 475.158030] Lustre: Skipped 1 previous similar message [ 479.520653] Lustre: server umount lustre-OST0000 complete [ 484.843560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 484.849815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.869418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.875953] Lustre: Skipped 1 previous similar message [ 490.229232] Lustre: server umount lustre-MDT0000 complete [ 490.471503] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.505553] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 495.204837] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832013 with bad export cookie 17241503651842492292 [ 495.219781] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 495.227564] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.806559] Lustre: server umount lustre-MDT0001 complete [ 502.729643] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:46:59 (1788832019) [ 524.947241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.381945] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.457582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.556163] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 530.916979] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.024272] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.906040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.854420] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 548.832780] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 555.360843] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.294674] Lustre: Failing over lustre-MDT0000 [ 558.567267] Lustre: server umount lustre-MDT0000 complete [ 559.585684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.604360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.625585] Lustre: Skipped 1 previous similar message [ 559.637938] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.319468] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.346336] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 576.479485] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832078/real 1788832078] req@ffff95cb43ce5500 x1875725873472384/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788832094 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 576.490197] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.395259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.722418] LustreError: 3639:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95cb43ca9180 x1875725873475200/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 586.757222] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.790590] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 591.802796] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 592.361723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 592.363271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 592.378731] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 601.627451] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 607.729194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.735968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.746312] Lustre: Skipped 1 previous similar message [ 607.753636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.758657] Lustre: Skipped 2 previous similar messages [ 611.835907] Lustre: server umount lustre-MDT0000 complete [ 616.012127] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832134 with bad export cookie 17241503651842493706 [ 616.022987] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.027232] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 616.470939] Lustre: server umount lustre-MDT0001 complete [ 625.752668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.137793] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.165081] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 626.252920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.262436] Lustre: Skipped 2 previous similar messages [ 630.592350] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 639.084232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.656493] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 652.742971] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.882279] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.611484] Lustre: Failing over lustre-MDT0000 [ 661.852545] Lustre: server umount lustre-MDT0000 complete [ 661.992836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.416886] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832182/real 1788832182] req@ffff95cc81a60a80 x1875725873502976/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788832198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.439571] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.918696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.660462] LustreError: 23997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.689801] LustreError: 23997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 692.060193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 696.837034] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 697.316571] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 697.326952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 697.349727] Lustre: Skipped 1 previous similar message [ 697.373214] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 707.176521] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 712.676542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 712.685470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.709599] Lustre: Skipped 1 previous similar message [ 712.716818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.720177] Lustre: Skipped 3 previous similar messages [ 718.140864] Lustre: server umount lustre-MDT0000 complete [ 722.432762] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832240 with bad export cookie 17241503651842494644 [ 722.451387] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.720380] Lustre: server umount lustre-MDT0001 complete [ 733.482626] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:50:50 (1788832250) [ 752.525240] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 764.926134] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 809.272536] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 822.569694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.800672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 822.829038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 822.902980] Lustre: lustre-MDT0000: new disk, initializing [ 822.968056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.973933] Lustre: Skipped 2 previous similar messages [ 822.986613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 828.883377] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 841.506674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.714625] Lustre: 29644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 842.075570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 846.694526] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 857.919085] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 863.961957] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 870.858884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.032526] Lustre: 31614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 873.109634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 873.137200] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 873.249063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 873.267225] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 877.922924] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 890.783567] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 902.326126] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 924.644176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.660469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.661996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.682957] Lustre: Skipped 2 previous similar messages [ 924.690907] Lustre: Skipped 2 previous similar messages [ 928.229957] Lustre: server umount lustre-OST0000 complete [ 939.052862] Lustre: server umount lustre-MDT0000 complete [ 940.009149] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.054801] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 943.411954] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832461 with bad export cookie 17241503651842495050 [ 943.426747] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.432288] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 943.772956] Lustre: server umount lustre-MDT0001 complete [ 958.861551] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 962.183939] Key type lgssc unregistered [ 962.513612] LNet: 33597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.527034] LNetError: 33597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.549569] LNet: Removed LNI 192.168.201.127@tcp [ 964.207299] Key type .llcrypt unregistered [ 964.218744] Key type ._llcrypt unregistered [ 995.513623] Key type ._llcrypt registered [ 995.533568] Key type .llcrypt registered [ 995.692748] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 997.403796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.663820] alg: No test for adler32 (adler32-zlib) [ 999.075927] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 999.486374] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1001.232396] Key type lgssc registered [ 1002.503161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.931905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.966631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.694813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.400223] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1021.930355] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.946610] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1027.019276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.043527] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.449625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.192464] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1041.444552] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.464863] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.873809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.439496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.419923] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1073.151230] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1074.779401] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.947130] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:56:49 (1788832609) [ 1095.582510] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1097.821590] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1105.011393] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:57:01 (1788832621) [ 1117.127880] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:57:14 (1788832634) [ 1118.041403] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1128.829759] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1128.836125] Lustre: Skipped 2 previous similar messages [ 1131.418797] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:57:27 (1788832647) [ 1134.281548] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1134.292660] Lustre: Skipped 2 previous similar messages [ 1144.594421] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1144.607280] Lustre: Skipped 1 previous similar message [ 1162.692041] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1162.700637] Lustre: Skipped 5 previous similar messages [ 1167.071218] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:58:03 (1788832683) [ 1169.999535] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1173.240687] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.064241] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:58:21 (1788832701) [ 1198.971136] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1198.980087] Lustre: Skipped 7 previous similar messages [ 1201.106261] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:58:38 (1788832718) [ 1221.603896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1221.605567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.628914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.639989] Lustre: Skipped 1 previous similar message [ 1224.491927] Lustre: server umount lustre-OST0000 complete [ 1231.841674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.863254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.875215] Lustre: Skipped 2 previous similar messages [ 1236.961110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.056759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.059555] Lustre: Skipped 2 previous similar messages [ 1243.105700] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1243.337738] Lustre: server umount lustre-MDT0000 complete [ 1246.180120] LustreError: 36206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.204317] LustreError: 36206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1247.542802] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832765 with bad export cookie 6800768674883394725 [ 1247.543758] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.554266] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1247.922863] Lustre: server umount lustre-MDT0001 complete [ 1261.101356] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1264.426170] Key type lgssc unregistered [ 1264.676795] LNet: 42872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.694238] LNetError: 42872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.765798] LNet: Removed LNI 192.168.201.127@tcp [ 1266.729347] Key type .llcrypt unregistered [ 1266.735083] Key type ._llcrypt unregistered [ 1286.480943] Key type ._llcrypt registered [ 1286.483911] Key type .llcrypt registered [ 1286.563086] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1306.552945] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1307.592181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.636781] alg: No test for adler32 (adler32-zlib) [ 1308.639925] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1308.883107] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1310.651395] Key type lgssc registered [ 1311.951606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.501104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1355.529640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.858768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1356.891464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1356.973104] Lustre: lustre-MDT0000: new disk, initializing [ 1357.073169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.096940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1362.456455] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1373.866455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.970467] Lustre: 46036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1374.040255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1374.049813] Lustre: Skipped 1 previous similar message [ 1374.208202] Lustre: lustre-MDT0001: new disk, initializing [ 1374.357739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.432987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1374.443019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1379.239566] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1389.147132] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.646865] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1402.335548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1402.563492] Lustre: lustre-OST0000: new disk, initializing [ 1402.566443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1402.570843] Lustre: 48006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1402.650632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.223564] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1411.667738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1411.676883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1411.798957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1419.313564] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.711557] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1425.035624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1430.164985] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1430.439553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1432.547699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.552947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1432.579275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.607598] Lustre: Skipped 1 previous similar message [ 1437.673868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.686823] Lustre: Skipped 1 previous similar message [ 1438.710596] Lustre: server umount lustre-OST0000 complete [ 1445.856618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.867920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.887223] Lustre: Skipped 1 previous similar message [ 1445.903192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.905269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.907763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.927783] Lustre: Skipped 1 previous similar message [ 1449.450165] Lustre: server umount lustre-MDT0000 complete [ 1453.653454] LustreError: 46029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832971 with bad export cookie 17862397305706269934 [ 1453.656827] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.671401] LustreError: 46029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1454.249270] Lustre: server umount lustre-MDT0001 complete [ 1465.168455] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:03:02 (1788832982) [ 1472.985612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.862432] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.889343] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1473.984643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.139300] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.380302] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1484.268528] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.379618] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.095186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.556903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1496.467321] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1506.389859] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1513.466218] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1521.523379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1521.951530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.930953] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1538.691957] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1550.827624] LustreError: 53010:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1552.521453] Lustre: Modifying parameter general.timeout=1 in log params [ 1553.744405] Lustre: Modifying parameter general.timeout=2 in log params [ 1555.829888] Lustre: Modifying parameter general.timeout=4 in log params [ 1555.833243] Lustre: Skipped 1 previous similar message [ 1560.027537] Lustre: Modifying parameter general.timeout=8 in log params [ 1560.061606] Lustre: Skipped 3 previous similar messages [ 1568.970895] Lustre: Modifying parameter general.timeout=15 in log params [ 1568.976485] Lustre: Skipped 6 previous similar messages [ 1586.349415] Lustre: Modifying parameter general.timeout=30 in log params [ 1586.358498] Lustre: Skipped 14 previous similar messages [ 1618.628858] Lustre: Modifying parameter general.timeout=60 in log params [ 1618.635479] Lustre: Skipped 29 previous similar messages [ 1683.401475] Lustre: Modifying parameter general.timeout=122 in log params [ 1683.407160] Lustre: Skipped 61 previous similar messages [ 1704.345235] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:07:01 (1788833221) [ 1707.523418] Lustre: 59968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1709.601294] Lustre: 60068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1712.320866] Lustre: 60166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1716.469785] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1716.476501] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1749.878771] Lustre: 61843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1749.887176] Lustre: 61843:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1749.914831] LustreError: 61843:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1755.005655] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:07:51 (1788833271) [ 1770.469600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.478854] Lustre: Skipped 1 previous similar message [ 1770.480986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.503943] Lustre: Skipped 2 previous similar messages [ 1775.912827] Lustre: server umount lustre-MDT0000 complete [ 1780.363983] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833298 with bad export cookie 17862397305706270816 [ 1780.365849] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.382216] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1780.705165] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.715136] LustreError: 51552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.966904] Lustre: server umount lustre-MDT0001 complete [ 1787.372976] Lustre: server umount lustre-OST0000 complete [ 1818.248499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.378891] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1818.393903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1818.401153] Lustre: Skipped 30 previous similar messages [ 1818.702682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.155271] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1836.597689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.765548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1836.847494] Lustre: 64377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.714968] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1854.782290] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1862.152178] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1870.959944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1871.212844] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1871.562106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.571584] Lustre: Skipped 1 previous similar message [ 1879.662668] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1888.480193] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1890.573518] Lustre: 66945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1903.082262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.084735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.098158] LustreError: Skipped 1 previous similar message [ 1903.106315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.114353] Lustre: Skipped 1 previous similar message [ 1903.147256] Lustre: Skipped 4 previous similar messages [ 1905.739836] Lustre: server umount lustre-OST0000 complete [ 1916.711087] Lustre: server umount lustre-MDT0000 complete [ 1918.439589] LustreError: 64384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.459690] LustreError: 64384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1921.057174] LustreError: 64368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833439 with bad export cookie 17862397305706274029 [ 1921.065394] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1921.068527] LustreError: 64368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1921.747286] Lustre: server umount lustre-MDT0001 complete [ 1935.274696] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1938.078840] Key type lgssc unregistered [ 1938.370107] LNet: 68156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.384769] LNetError: 68156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.414029] LNet: Removed LNI 192.168.201.127@tcp [ 1939.601408] Key type .llcrypt unregistered [ 1939.603680] Key type ._llcrypt unregistered [ 1973.771422] Key type ._llcrypt registered [ 1973.776183] Key type .llcrypt registered [ 1973.896876] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1975.290226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.414126] alg: No test for adler32 (adler32-zlib) [ 1976.861445] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1977.270636] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1979.039492] Key type lgssc registered [ 1980.951952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1990.988665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.024951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.649354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.830150] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2001.895278] LustreError: 69566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.934699] LustreError: 69566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2007.019605] LustreError: 69567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.583578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.204745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.194339] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2026.206842] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2033.826795] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.828220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.549271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.033850] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2060.894799] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.532374] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2086.881223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2086.895177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.910710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.917289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.934528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.952441] Lustre: Skipped 1 previous similar message [ 2087.965407] Lustre: Skipped 1 previous similar message [ 2089.818295] Lustre: server umount lustre-MDT0000 complete [ 2093.029229] LustreError: 70297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.056557] LustreError: 70297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2094.282274] LustreError: 72254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833612 with bad export cookie 15465839750756189287 [ 2094.294308] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.298981] LustreError: 72254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2094.716487] Lustre: server umount lustre-MDT0001 complete [ 2111.084449] Lustre: server umount lustre-OST0000 complete [ 2143.310420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.576396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2143.615847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2144.031098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.538731] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2161.117201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.312876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2161.366393] Lustre: 74441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2161.871591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.866654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2179.170602] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2186.989857] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2196.493050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.658661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2197.242090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.549764] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2216.544398] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2218.658719] Lustre: 77021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2221.067946] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2231.278062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2231.280942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.285950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.313936] Lustre: Skipped 1 previous similar message [ 2234.160475] Lustre: server umount lustre-OST0000 complete [ 2241.507556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.523687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.532907] Lustre: Skipped 1 previous similar message [ 2241.546667] Lustre: Skipped 2 previous similar messages [ 2246.634292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.646529] Lustre: Skipped 1 previous similar message [ 2253.279743] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2253.588963] Lustre: server umount lustre-MDT0000 complete [ 2257.888432] LustreError: 74452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.912887] LustreError: 74452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2258.465766] LustreError: 74434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833776 with bad export cookie 15465839750756190134 [ 2258.475171] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.486294] LustreError: 74434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2258.899648] Lustre: server umount lustre-MDT0001 complete [ 2270.609775] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2274.001894] Key type lgssc unregistered [ 2274.280368] LNet: 78232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.293545] LNetError: 78232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.368042] LNet: Removed LNI 192.168.201.127@tcp [ 2276.117082] Key type .llcrypt unregistered [ 2276.119295] Key type ._llcrypt unregistered [ 2308.248204] Key type ._llcrypt registered [ 2308.249528] Key type .llcrypt registered [ 2308.334808] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2309.226384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.521174] alg: No test for adler32 (adler32-zlib) [ 2310.516732] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2310.916980] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2312.783212] Key type lgssc registered [ 2314.670841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.246028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2326.285661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.313075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.709284] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2338.786178] LustreError: 79643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.819057] LustreError: 79643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2343.910710] LustreError: 79644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.052527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.045877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.283594] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2363.043753] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2370.157200] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2381.759494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.827149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.676139] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2401.712912] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2409.373962] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2410.511248] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2412.699964] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2412.705192] Lustre: Skipped 1 previous similar message [ 2414.964726] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2414.974034] Lustre: Skipped 1 previous similar message [ 2419.036453] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2419.044471] Lustre: Skipped 3 previous similar messages [ 2427.119464] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2427.126348] Lustre: Skipped 7 previous similar messages [ 2443.394639] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2443.404217] Lustre: Skipped 14 previous similar messages [ 2476.295140] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2476.303947] Lustre: Skipped 28 previous similar messages [ 2532.653163] Lustre: 87761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2536.596477] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:20:52 (1788834052) [ 2540.314988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2542.787858] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:20:59 (1788834059) [ 2550.756587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2550.775533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.804247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.802522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.816551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.852575] Lustre: Skipped 1 previous similar message [ 2555.043611] Lustre: server umount lustre-OST0000 complete [ 2562.538293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2562.550510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.585947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.166498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.169370] Lustre: Skipped 2 previous similar messages [ 2573.281043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.284555] Lustre: Skipped 1 previous similar message [ 2575.327407] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2575.666559] Lustre: server umount lustre-MDT0000 complete [ 2578.404620] LustreError: 79639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.433477] LustreError: 79639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2580.108493] LustreError: 81501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834098 with bad export cookie 8936685477704840009 [ 2580.115174] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.132560] LustreError: 81501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2580.487034] Lustre: server umount lustre-MDT0001 complete [ 2593.627544] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2597.261399] Key type lgssc unregistered [ 2597.750937] LNet: 89013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.759267] LNetError: 89013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.788445] LNet: Removed LNI 192.168.201.127@tcp [ 2598.787576] Key type .llcrypt unregistered [ 2598.790054] Key type ._llcrypt unregistered [ 2604.583727] Key type ._llcrypt registered [ 2604.585755] Key type .llcrypt registered [ 2604.698490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2610.724827] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules [ 2612.396339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2612.476918] alg: No test for adler32 (adler32-zlib) [ 2613.738483] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2614.119136] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2615.903818] Key type lgssc registered [ 2617.939949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2619.823771] LustreError: 90261:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2642.597641] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2642.828389] LustreError: 90261:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2654.618655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2654.642638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.378467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.044935] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2665.959030] LustreError: 91287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.980174] LustreError: 91287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2671.080436] LustreError: 91288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.492590] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:23:10 (1788834190) [ 2676.205349] LustreError: 91287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.635890] LustreError: 91319:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2680.654960] LustreError: 91319:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2680.666796] LustreError: 91319:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2681.367884] Lustre: server umount lustre-MDT0000 complete [ 2698.517844] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2702.264772] Key type lgssc unregistered [ 2702.717705] LNet: 92811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.729726] LNetError: 92811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.761061] LNet: Removed LNI 192.168.201.127@tcp [ 2703.925121] Key type .llcrypt unregistered [ 2703.930599] Key type ._llcrypt unregistered [ 2729.869413] Key type ._llcrypt registered [ 2729.876922] Key type .llcrypt registered [ 2730.089353] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2731.819675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.897430] alg: No test for adler32 (adler32-zlib) [ 2733.135981] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2733.498233] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2735.247222] Key type lgssc registered [ 2737.059589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2746.660875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2746.674862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.070146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.447889] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2758.642814] LustreError: 94027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.685450] LustreError: 94027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2763.746412] LustreError: 94026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.235436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.200381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.647982] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2779.932432] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2787.344535] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2797.255838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.329075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.748426] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2816.368992] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2874.075862] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:26:31 (1788834391) [ 2875.988816] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2878.098729] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:26:35 (1788834395) [ 2885.625995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.637964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.644773] Lustre: Skipped 2 previous similar messages [ 2885.666725] Lustre: Skipped 2 previous similar messages [ 2890.559912] Lustre: server umount lustre-MDT0000 complete [ 2890.721411] LustreError: 94026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.749723] LustreError: 94026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2895.803789] LustreError: 94008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834414 with bad export cookie 2993311879252927005 [ 2895.823124] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.828749] LustreError: 94008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2895.860094] LustreError: 94022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.879799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.923337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2896.577415] Lustre: server umount lustre-MDT0001 complete [ 2903.547098] Lustre: server umount lustre-OST0000 complete [ 2918.732525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.372860] LustreError: 98325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.616080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.406623] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2929.138041] LustreError: 98325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.175470] LustreError: 98325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2935.281560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.017340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2941.582401] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2950.885378] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2958.135247] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2973.817606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.975162] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2973.979966] LustreError: 98315:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2973.999554] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2974.006889] LustreError: 100411:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2974.011799] LustreError: 100411:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2974.016070] LustreError: 100411:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2974.081728] Lustre: server umount lustre-OST0000 complete [ 2974.089771] LustreError: 100411:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2984.113491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.361844] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2984.366903] LustreError: 98315:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2984.389883] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2984.397732] LustreError: 100877:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2984.405771] LustreError: 100877:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2984.411052] LustreError: 100877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2984.479958] Lustre: server umount lustre-OST0000 complete [ 2984.482056] LustreError: 100877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2996.011886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.393285] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2996.411113] Lustre: Found index 0 for lustre-OST0000, updating log [ 2996.419744] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2996.533343] Lustre: lustre-OST0000: new disk, initializing [ 2996.537403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2996.544203] Lustre: 101414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2996.925566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.108523] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3010.622812] Lustre: Failing over lustre-OST0000 [ 3010.751392] Lustre: server umount lustre-OST0000 complete [ 3017.699190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.711339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.723369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.257034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.258112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.220414] Lustre: server umount lustre-MDT0000 complete [ 3025.377619] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.405307] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3025.722618] LustreError: 98305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834543 with bad export cookie 2993311879252929749 [ 3025.728462] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3025.731960] LustreError: 98305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3035.495395] Lustre: server umount lustre-MDT0001 complete [ 3046.087691] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:29:22 (1788834562) [ 3066.127822] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3077.277469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.842319] LustreError: 103762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.956760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.377445] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3093.100787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.198841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3101.377910] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3108.719401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.582995] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3117.312497] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3117.806107] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3117.831814] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3117.832795] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3126.650515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.727789] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3126.795537] Lustre: lustre-OST0001: new disk, initializing [ 3126.798590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3126.805208] Lustre: 106083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3127.038575] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.043665] Lustre: Skipped 2 previous similar messages [ 3134.700378] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3136.700902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3136.708904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3136.828712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3142.616435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3148.956182] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3154.912334] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3154.920688] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.937317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3154.950246] Lustre: Skipped 1 previous similar message [ 3160.274135] Lustre: server umount lustre-MDT0001 complete [ 3162.601543] LustreError: 103763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.623147] LustreError: 103763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3171.355467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.524691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3171.534522] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3171.642825] Lustre: 103749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3173.547488] LustreError: 107626:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3173.557102] LustreError: 107626:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3173.570996] LustreError: 103754:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3173.672294] LustreError: 107635:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.127@tcp: cfg command failed: rc = -17 [ 3173.677728] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3173.677728] [ 3177.983663] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3179.125725] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3191.068717] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:31:47 (1788834707) [ 3212.551793] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3234.355257] Lustre: 109706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3260.520370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3265.006341] Lustre: 110842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3361.186696] Lustre: server umount lustre-MDT0000 complete [ 3361.760161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3361.771927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.793566] Lustre: Skipped 2 previous similar messages [ 3361.809652] LustreError: 103763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.829083] LustreError: 103763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3365.239844] LustreError: 103742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834883 with bad export cookie 2993311879252930750 [ 3365.240979] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.255101] LustreError: 103742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3365.782062] Lustre: server umount lustre-MDT0001 complete [ 3381.520460] Lustre: server umount lustre-OST0000 complete [ 3383.391574] Lustre: 93337:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834885/real 1788834885] req@ffff95cc7c197b80 x1875728663951488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3383.438814] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.464446] Lustre: Skipped 3 previous similar messages [ 3386.090503] Lustre: server umount lustre-OST0001 complete [ 3392.031299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3403.378366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3412.351514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3423.334042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3446.646274] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3458.710458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.746875] Lustre: lustre-MDT0000: reset Object Index mappings [ 3459.660772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.671326] Lustre: Skipped 1 previous similar message [ 3465.884199] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3476.368837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.402658] Lustre: lustre-MDT0001: reset Object Index mappings [ 3483.269539] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3486.654340] Lustre: 115939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3493.492530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.167327] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3502.401321] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3505.475666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3511.898825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.951498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3517.972085] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3521.528897] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3531.706704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3537.339638] Lustre: 117817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3595.411824] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:38:32 (1788835112) [ 3620.345134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3620.349638] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.376824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.385241] Lustre: Skipped 3 previous similar messages [ 3620.399630] Lustre: Skipped 7 previous similar messages [ 3634.143430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3634.484697] Lustre: server umount lustre-MDT0000 complete [ 3635.695680] LustreError: 117853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.707892] LustreError: 117853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 3638.104347] LustreError: 114773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835156 with bad export cookie 2993311879252977622 [ 3638.110079] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.121612] LustreError: 114773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3640.815666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3640.828081] Lustre: Skipped 9 previous similar messages [ 3644.829748] Lustre: server umount lustre-MDT0001 complete [ 3655.714360] Lustre: server umount lustre-OST0000 complete [ 3666.572458] Lustre: server umount lustre-OST0001 complete [ 3675.188930] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3687.449920] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3728.798523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.040746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3729.078747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3729.149527] Lustre: lustre-MDT0000: new disk, initializing [ 3729.211934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.218626] Lustre: Skipped 3 previous similar messages [ 3729.233875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3734.405285] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3741.965693] Lustre: server umount lustre-MDT0000 complete [ 3756.210325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.751274] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3756.769607] Lustre: 122094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3756.781239] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3762.963667] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3769.090090] Lustre: server umount lustre-MDT0000 complete [ 3781.253926] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:41:37 (1788835297) [ 3783.094388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3785.099484] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:41:42 (1788835302) [ 3805.299764] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3817.241279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.373597] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3833.485863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3833.626219] Lustre: 124148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3833.693986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3833.699140] Lustre: Skipped 1 previous similar message [ 3833.802404] Lustre: lustre-MDT0001: new disk, initializing [ 3833.933736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3833.947663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3838.828262] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3844.123847] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3851.503471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.830855] Lustre: lustre-OST0000: new disk, initializing [ 3851.834332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3851.841152] Lustre: 125727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3853.127913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3853.141988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3853.320656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3859.388910] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3871.549995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3871.792532] Lustre: lustre-OST0001: new disk, initializing [ 3871.796626] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3871.810844] Lustre: 126599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3873.445299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3873.471464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3873.640349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3879.527834] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3890.102984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3895.310751] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4347.696123] Lustre: 133025:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4347.710770] Lustre: 133025:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4347.717900] Lustre: 133025:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4347.729336] Lustre: 133025:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4347.739628] Lustre: 133025:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4347.745694] Lustre: 133025:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4670.553195] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:56:27 (1788836187) [ 4688.865372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.873438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.900328] Lustre: Skipped 2 previous similar messages [ 4688.919543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.933158] Lustre: Skipped 2 previous similar messages [ 4692.672663] Lustre: server umount lustre-MDT0000 complete [ 4696.035664] LustreError: 127618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.058896] LustreError: 127618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4696.206982] LustreError: 124886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836214 with bad export cookie 2993311879253046012 [ 4696.217459] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4696.224547] LustreError: 124886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4701.153165] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.156653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4701.189219] Lustre: Skipped 4 previous similar messages [ 4701.204871] Lustre: Skipped 5 previous similar messages [ 4702.969287] Lustre: server umount lustre-MDT0001 complete [ 4723.167154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4723.446952] Lustre: server umount lustre-OST0000 complete [ 4735.171394] Lustre: server umount lustre-OST0001 complete [ 4742.840053] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4755.939452] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4800.513954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.948976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4801.010407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4801.137657] Lustre: lustre-MDT0000: new disk, initializing [ 4801.272488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.279995] Lustre: Skipped 5 previous similar messages [ 4801.306705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4805.687496] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4818.186293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.308475] Lustre: 136091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4818.354372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4818.360540] Lustre: Skipped 1 previous similar message [ 4818.440395] Lustre: lustre-MDT0001: new disk, initializing [ 4818.561197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4818.569455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4824.067249] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4835.240629] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4841.800280] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4850.580261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4850.951913] Lustre: lustre-OST0000: new disk, initializing [ 4850.957492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4850.969030] Lustre: 138060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4851.092080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.095173] Lustre: Skipped 1 previous similar message [ 4852.298428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4852.318307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4852.477432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4859.438047] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4870.716786] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4882.732494] Lustre: lustre-MDD0000: changelog on [ 4885.960770] Lustre: lustre-MDD0001: changelog on [ 6922.396177] Lustre: lustre-MDD0001: changelog off [ 6924.514729] Lustre: 138039:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6925.522563] Lustre: lustre-MDD0000: changelog off [ 6936.033466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6936.045106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.058498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.550050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.669230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.677050] Lustre: Skipped 2 previous similar messages [ 6946.271323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6946.441915] Lustre: server umount lustre-OST0000 complete [ 6951.921593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6951.928239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.937893] Lustre: Skipped 2 previous similar messages [ 6956.804242] Lustre: server umount lustre-MDT0000 complete [ 6957.029986] LustreError: 136101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.051686] LustreError: 136101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6960.717947] LustreError: 136081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838478 with bad export cookie 2993311879253868393 [ 6960.739451] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.013137] Lustre: server umount lustre-MDT0001 complete [ 6973.123883] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 6975.777621] Key type lgssc unregistered [ 6976.181824] LNet: 147228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6976.194763] LNetError: 147228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6976.227439] LNet: Removed LNI 192.168.201.127@tcp [ 6977.039553] Key type .llcrypt unregistered [ 6977.044113] Key type ._llcrypt unregistered [ 7000.975665] Key type ._llcrypt registered [ 7000.978288] Key type .llcrypt registered [ 7001.146427] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 7019.526740] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7021.037698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7021.071972] alg: No test for adler32 (adler32-zlib) [ 7022.177890] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 7022.377112] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 7024.095177] Key type lgssc registered [ 7025.102167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7063.872595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7063.893350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.236400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7065.262376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7065.372582] Lustre: lustre-MDT0000: new disk, initializing [ 7065.492977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.508992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7070.869882] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7082.670752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.872877] Lustre: 150430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7082.963975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7082.970171] Lustre: Skipped 1 previous similar message [ 7083.077984] Lustre: lustre-MDT0001: new disk, initializing [ 7083.210692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7083.265497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7083.282213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7088.628395] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7099.179119] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.645274] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.808934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.297635] Lustre: lustre-OST0000: new disk, initializing [ 7114.304263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7114.312676] Lustre: 152399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7114.418407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.547182] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7123.520066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7123.534685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7123.677634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7133.449476] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.243436] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_module obdecho/obdecho [ 7141.555555] LustreError: 153401:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7141.563759] LustreError: 153401:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7143.474589] LustreError: 153498:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7154.658250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.672476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.674815] Lustre: Skipped 1 previous similar message [ 7154.685429] Lustre: Skipped 2 previous similar messages [ 7159.777956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7162.849818] Lustre: server umount lustre-MDT0000 complete [ 7166.937227] LustreError: 150422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838685 with bad export cookie 9104291328100317833 [ 7166.944786] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.322698] Lustre: server umount lustre-MDT0001 complete [ 7181.986117] Lustre: server umount lustre-OST0000 complete [ 7203.199490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 7216.750258] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7258.668825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.033391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7259.065500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7259.155327] Lustre: lustre-MDT0000: new disk, initializing [ 7259.331438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.390464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7264.624796] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7276.717034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.829673] Lustre: 157216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7276.901328] LustreError: 157226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.923626] LustreError: 157226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7276.933810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7276.935951] Lustre: Skipped 1 previous similar message [ 7276.993237] Lustre: lustre-MDT0001: new disk, initializing [ 7277.069599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.123479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7277.139539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7282.272545] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7294.453643] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7301.265550] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7309.161516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.460956] Lustre: lustre-OST0000: new disk, initializing [ 7309.463947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7309.478761] Lustre: 159183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7309.620197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.101552] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7317.579925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7317.593634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7317.738524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7326.173916] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7334.082532] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7334.434312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7341.147280] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7341.406470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7348.713885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7348.715161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.721398] LustreError: Skipped 1 previous similar message [ 7348.722499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7348.740994] Lustre: Skipped 2 previous similar messages [ 7348.751934] Lustre: Skipped 3 previous similar messages [ 7349.968549] Lustre: server umount lustre-OST0000 complete [ 7358.944068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7358.947259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.959357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7358.973591] Lustre: Skipped 1 previous similar message [ 7358.985428] Lustre: Skipped 1 previous similar message [ 7360.413272] Lustre: server umount lustre-MDT0000 complete [ 7364.065963] LustreError: 157226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.095846] LustreError: 157226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7364.663363] LustreError: 157208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838882 with bad export cookie 9104291328100318792 [ 7364.682271] LustreError: 157208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7364.692395] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.119290] Lustre: server umount lustre-MDT0001 complete [ 7369.790976] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:41:26 (1788838886) [ 7371.591294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7373.341139] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:41:30 (1788838890) [ 7382.019856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.333450] LustreError: 161599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.418090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.243365] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7387.625645] LustreError: 161600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7392.739725] LustreError: 161599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.872866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.867071] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7412.365186] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7419.669965] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7427.848837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.206368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.208822] Lustre: Skipped 1 previous similar message [ 7434.245604] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7442.493531] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.392830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.402511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7463.420890] Lustre: Skipped 2 previous similar messages [ 7463.445231] Lustre: Skipped 3 previous similar messages [ 7465.320508] Lustre: server umount lustre-MDT0000 complete [ 7468.500933] LustreError: 163954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.511770] LustreError: 161580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838987 with bad export cookie 9104291328100319674 [ 7469.539875] LustreError: 161580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7469.559900] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.636957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.664470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7476.026575] Lustre: server umount lustre-MDT0001 complete [ 7482.713143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7495.654233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.799349] LustreError: 165463:0:(llog.c:1655:llog_backup()) MGC192.168.201.127@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7495.809828] Lustre: 165463:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.127@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7499.300659] LustreError: 165468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.331963] LustreError: 165468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7533.957522] LustreError: 165469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.979668] LustreError: 165469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7556.575729] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7556.582673] Lustre: 165463:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7597.567521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7600.588602] LustreError: 165488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.610261] LustreError: 165488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 7602.664524] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7602.947363] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7613.311199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.928925] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7e58f0b9ef16daf0 [ 7613.943451] Lustre: MGC192.168.201.127@tcp: Connection restored to 0@lo (at 0@lo) [ 7619.562802] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7619.978468] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7628.024980] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7629.848509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7636.842749] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7638.605812] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7660.512129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7660.516750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.528420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.544256] Lustre: Skipped 1 previous similar message [ 7660.555643] Lustre: Skipped 4 previous similar messages [ 7661.268686] Lustre: server umount lustre-MDT0000 complete [ 7664.988992] LustreError: 161580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839183 with bad export cookie 9104291328100326128 [ 7665.002367] LustreError: 161580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7665.006100] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7665.507485] Lustre: server umount lustre-MDT0001 complete [ 7679.608678] Lustre: server umount lustre-OST0000 complete [ 7701.663770] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 7712.072525] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7749.961174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.309145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7750.337736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7750.437481] Lustre: lustre-MDT0000: new disk, initializing [ 7750.537309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.541881] Lustre: Skipped 1 previous similar message [ 7750.554428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7754.845846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7767.876603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.106507] Lustre: 170585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7768.714803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7775.080793] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7786.371634] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7794.558249] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7804.000985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7804.276290] Lustre: lustre-OST0000: new disk, initializing [ 7804.285462] Lustre: Skipped 1 previous similar message [ 7804.294859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7804.305066] Lustre: Skipped 2 previous similar messages [ 7804.324616] Lustre: 172557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7811.681078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7811.697231] Lustre: Skipped 1 previous similar message [ 7811.704549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7811.892879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7813.145933] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7824.308541] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7830.398561] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7830.682816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7836.549102] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7836.855985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7842.788774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7842.789318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.824853] Lustre: Skipped 1 previous similar message [ 7842.838881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7845.423524] Lustre: server umount lustre-OST0000 complete [ 7850.464928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7850.473714] LustreError: Skipped 1 previous similar message [ 7864.287541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.681521] Lustre: server umount lustre-MDT0000 complete [ 7868.394547] LustreError: 170590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.424949] LustreError: 170590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7868.893173] LustreError: 172556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839387 with bad export cookie 9104291328100342963 [ 7868.898090] LustreError: 172556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7868.903154] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7869.250196] Lustre: server umount lustre-MDT0001 complete [ 7879.737901] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:49:56 (1788839396) [ 7888.225733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7895.011567] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7903.932303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.423799] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7917.893630] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.651747] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7933.395563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7942.386522] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7951.743322] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.605260] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:51:32 (1788839492) [ 7984.104420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.113899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7984.116561] Lustre: Skipped 4 previous similar messages [ 7984.120093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7984.120101] Lustre: Skipped 8 previous similar messages [ 7987.833555] Lustre: server umount lustre-OST0000 complete [ 7999.133458] Lustre: server umount lustre-MDT0000 complete [ 8004.053728] LustreError: 174953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839522 with bad export cookie 9104291328100343845 [ 8004.075213] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8004.081489] LustreError: 174953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8004.692805] Lustre: server umount lustre-MDT0001 complete [ 8018.782151] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8022.154537] Key type lgssc unregistered [ 8022.426957] LNet: 178726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8022.439297] LNetError: 178726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8023.467901] LNet: Removed LNI 192.168.201.127@tcp [ 8024.838980] Key type .llcrypt unregistered [ 8024.843364] Key type ._llcrypt unregistered [ 8047.975088] Key type ._llcrypt registered [ 8047.976488] Key type .llcrypt registered [ 8048.108504] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8049.976301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8050.044749] alg: No test for adler32 (adler32-zlib) [ 8051.258661] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 8051.598389] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8053.271537] Key type lgssc registered [ 8054.784968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8064.382801] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8064.437586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.031469] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8085.157448] LustreError: 179972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [ 8090.271716] LustreError: 179972:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8090.308727] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8090.330493] LustreError: MGC192.168.201.127@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8090.350906] LustreError: 179972:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8090.360636] LustreError: 179972:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 8090.376740] LustreError: 179972:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8090.482908] Lustre: server umount lustre-OST0000 complete [ 8090.492360] LustreError: 179972:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8101.832957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.454918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.735339] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8113.637631] LustreError: 180473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.664416] LustreError: 180473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8118.761160] LustreError: 180472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.115872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.466054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.249224] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8135.368623] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8143.015456] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.048655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8151.233193] LustreError: 182289:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8151.243555] Lustre: 182289:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.127@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8151.473921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.210699] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8169.743977] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.116176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.125660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8176.143724] Lustre: Skipped 2 previous similar messages [ 8176.149883] Lustre: Skipped 1 previous similar message [ 8179.024736] Lustre: server umount lustre-MDT0000 complete [ 8181.226038] LustreError: 180473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8181.255230] LustreError: 180473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8183.649353] LustreError: 180452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839701 with bad export cookie 110965874995450777 [ 8183.651996] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8183.665068] LustreError: 180452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8184.142449] Lustre: server umount lustre-MDT0001 complete [ 8199.648561] Lustre: server umount lustre-OST0000 complete [ 8213.104714] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 8229.756774] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8277.680361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.014235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8278.036514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8278.117840] Lustre: lustre-MDT0000: new disk, initializing [ 8278.221077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.251129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8285.300255] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8300.264238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.511845] Lustre: 185839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8300.571132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8300.576889] Lustre: Skipped 1 previous similar message [ 8300.709796] Lustre: lustre-MDT0001: new disk, initializing [ 8300.856520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.936077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8300.949763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8305.452927] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8318.836904] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8325.336578] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8334.468785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.847501] Lustre: lustre-OST0000: new disk, initializing [ 8334.850778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8334.856263] Lustre: 187806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.015779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.616131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8342.626236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8342.765232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8343.267684] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8355.712262] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.885756] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8362.313957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8369.605550] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8369.979819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8373.728983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8373.731405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8373.736853] LustreError: Skipped 1 previous similar message [ 8373.738740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8373.738747] Lustre: Skipped 1 previous similar message [ 8373.786295] Lustre: Skipped 1 previous similar message [ 8378.849265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.857276] Lustre: Skipped 2 previous similar messages [ 8379.897390] Lustre: server umount lustre-OST0000 complete [ 8388.070131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8388.086850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.113212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.428516] Lustre: server umount lustre-MDT0000 complete [ 8394.209266] LustreError: 185843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.231062] LustreError: 185843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8396.167057] LustreError: 185829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839914 with bad export cookie 110965874995451491 [ 8396.168084] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8396.176275] LustreError: 185829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8396.917134] Lustre: server umount lustre-MDT0001 complete [ 8408.232643] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:58:45 (1788839925) [ 8432.211396] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8435.787486] Key type lgssc unregistered [ 8436.163191] LNet: 190755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.171824] LNetError: 190755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8436.198834] LNet: Removed LNI 192.168.201.127@tcp [ 8437.496526] Key type .llcrypt unregistered [ 8437.498747] Key type ._llcrypt unregistered [ 8459.532953] Key type ._llcrypt registered [ 8459.538181] Key type .llcrypt registered [ 8459.668952] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8460.997370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8461.028207] alg: No test for adler32 (adler32-zlib) [ 8462.295150] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 8462.546295] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8464.255240] Key type lgssc registered [ 8465.766385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8475.807816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8475.841926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.484370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8483.790959] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8486.883675] LustreError: 191973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.907164] LustreError: 191973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8492.008338] LustreError: 191974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.768822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8495.355544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8500.989739] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8510.170139] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8516.460168] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8525.313908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8525.629272] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8525.632428] LustreError: 193840:0:(llog.c:1696:llog_backup()) MGC192.168.201.127@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8525.649639] Lustre: 193840:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.127@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8525.825586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.379459] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8544.406787] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8550.375228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.390230] Lustre: Skipped 2 previous similar messages [ 8550.398532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.406905] Lustre: Skipped 2 previous similar messages [ 8553.236250] Lustre: server umount lustre-MDT0000 complete [ 8555.489691] LustreError: 192703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.532568] LustreError: 192703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8557.848896] LustreError: 191955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840076 with bad export cookie 1297408554816624877 [ 8557.857937] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8557.870398] LustreError: 191955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8558.348101] Lustre: server umount lustre-MDT0001 complete [ 8573.057023] Lustre: server umount lustre-OST0000 complete [ 8588.942724] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 8606.830862] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8653.273965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.441794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8653.462594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8653.522542] Lustre: lustre-MDT0000: new disk, initializing [ 8653.666349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.689569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8659.695238] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8675.225880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8675.507718] Lustre: 197401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8675.620239] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8675.626492] Lustre: Skipped 1 previous similar message [ 8675.629910] LustreError: 198184:0:(llog.c:1696:llog_backup()) MGC192.168.201.127@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8675.636714] LustreError: 198184:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8675.641632] Lustre: 198184:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.127@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8675.664388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8675.668113] Lustre: Skipped 1 previous similar message [ 8675.783140] Lustre: lustre-MDT0001: new disk, initializing [ 8675.914747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8675.979659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8675.995665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8681.728969] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8692.656262] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8699.376968] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8707.682701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8708.136659] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8708.144245] Lustre: Skipped 1 previous similar message [ 8708.150979] LustreError: 199346:0:(llog.c:1696:llog_backup()) MGC192.168.201.127@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8708.157673] LustreError: 199346:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8708.163608] Lustre: 199346:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.127@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8708.184931] Lustre: 199346:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8708.218702] Lustre: lustre-OST0000: new disk, initializing [ 8708.221965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8708.230354] Lustre: 199370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8708.343163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.341503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8715.360264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8715.472938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8715.876586] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8726.119736] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8731.562408] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8731.830433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8737.906845] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8738.239720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8741.351134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.356112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8741.385194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8746.471286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8746.477429] Lustre: Skipped 1 previous similar message [ 8746.700845] Lustre: server umount lustre-OST0000 complete [ 8751.596140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.614753] Lustre: Skipped 1 previous similar message [ 8751.619530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8751.623249] Lustre: Skipped 1 previous similar message [ 8756.708185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.719732] Lustre: Skipped 2 previous similar messages [ 8756.847641] Lustre: server umount lustre-MDT0000 complete [ 8761.534909] LustreError: 198196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840279 with bad export cookie 1297408554816625591 [ 8761.551676] LustreError: 198196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8761.552057] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.873302] Lustre: server umount lustre-MDT0001 complete [ 8772.211990] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:04:49 (1788840289) [ 8793.682472] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8804.298675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.879338] LustreError: 202498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.964176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.966506] LustreError: 202499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.048421] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8820.203166] LustreError: 202498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.218302] LustreError: 202498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8821.650736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.411973] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8831.869472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8841.140294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.455717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.460264] Lustre: Skipped 1 previous similar message [ 8850.073910] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8861.751706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8862.038219] Lustre: lustre-OST0001: new disk, initializing [ 8862.045458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8862.053213] Lustre: 204810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.055985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8868.081019] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8868.165648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8869.889887] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8882.565874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8891.016284] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8893.391955] Lustre: DEBUG MARKER: ADD OST3 [ 8899.872554] loop: module loaded [ 8900.034490] loop0: detected capacity change from 0 to 204800000 [ 8900.398665] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8900.410729] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8900.427753] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8908.508906] Lustre: DEBUG MARKER: STOP OST3 [ 8908.640274] loop0: detected capacity change from 0 to 204800000 [ 8915.156389] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8915.512928] Lustre: lustre-OST0003: new disk, initializing [ 8915.532029] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8915.549651] Lustre: 206840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8915.822549] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8915.832772] Lustre: Skipped 1 previous similar message [ 8917.558045] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8917.717214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8917.728575] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8917.784161] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8919.593115] LustreError: 206991:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8919.703601] Lustre: Failing over lustre-OST0003 [ 8919.901363] Lustre: server umount lustre-OST0003 complete [ 8922.635116] LustreError: 204368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.107160] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.124920] Lustre: Skipped 1 previous similar message [ 8923.132088] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8924.442971] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8934.350383] Lustre: DEBUG MARKER: START OST3 again [ 8943.333539] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8943.908963] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8945.083824] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8945.232097] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8945.242563] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8951.874224] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8975.252743] Lustre: Failing over lustre-OST0003 [ 8975.329752] Lustre: server umount lustre-OST0003 complete [ 8976.354889] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8976.361529] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8976.364394] LustreError: Skipped 1 previous similar message [ 8976.371300] LustreError: 203986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.392614] Lustre: Skipped 2 previous similar messages [ 8976.432036] LustreError: 203986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8996.320624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8996.330244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.346473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.573882] Lustre: server umount lustre-MDT0000 complete [ 9007.315321] LustreError: 202480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840525 with bad export cookie 1297408554816626473 [ 9007.341420] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9012.707513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9012.716177] Lustre: Skipped 7 previous similar messages [ 9014.102112] Lustre: server umount lustre-MDT0001 complete [ 9020.837599] Lustre: server umount lustre-OST0000 complete [ 9025.628502] Lustre: server umount lustre-OST0001 complete [ 9036.688163] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9047.855697] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9101.027743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.468916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9101.520678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9101.636502] Lustre: lustre-MDT0000: new disk, initializing [ 9101.724136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9101.734995] Lustre: Skipped 1 previous similar message [ 9101.752720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9108.129792] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9121.561841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.700368] Lustre: 212278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9121.760487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9121.764205] Lustre: Skipped 1 previous similar message [ 9121.830386] Lustre: lustre-MDT0001: new disk, initializing [ 9122.059567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9122.071361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9127.354574] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9137.640385] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9143.907463] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.451991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.984538] Lustre: lustre-OST0000: new disk, initializing [ 9153.987047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9154.004455] Lustre: 214247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9155.087176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9155.108492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9155.290716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9161.951974] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9172.196426] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9177.634475] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9177.923815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9183.660556] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9183.887560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9191.408192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9191.408498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.426650] Lustre: Skipped 5 previous similar messages [ 9191.448388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9192.612340] Lustre: server umount lustre-OST0000 complete [ 9211.364069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9211.587890] Lustre: server umount lustre-MDT0000 complete [ 9215.968414] LustreError: 213696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9216.007300] LustreError: 213696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9216.154155] LustreError: 212269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840734 with bad export cookie 1297408554816628769 [ 9216.155796] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.163125] LustreError: 212269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9216.585731] Lustre: server umount lustre-MDT0001 complete [ 9220.983525] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:12:18 (1788840738) [ 9238.586965] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9249.562441] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9288.982477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.276283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9289.351518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9289.445762] Lustre: lustre-MDT0000: new disk, initializing [ 9289.594688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9294.233847] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9305.625657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.781312] Lustre: 218859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9306.187944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9311.773784] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9321.715524] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9328.886719] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9337.022552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9337.479805] Lustre: 220832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9338.914688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9339.136910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9346.619995] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9357.588065] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9363.405412] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9363.716813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9369.868131] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9370.135372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9375.202404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9375.207400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.216823] LustreError: Skipped 3 previous similar messages [ 9375.229542] Lustre: Skipped 3 previous similar messages [ 9375.232587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9375.232593] Lustre: Skipped 8 previous similar messages [ 9378.528105] Lustre: server umount lustre-OST0000 complete [ 9389.560107] Lustre: server umount lustre-MDT0000 complete [ 9390.573746] LustreError: 218872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9390.595878] LustreError: 218872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9393.923047] LustreError: 218853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840912 with bad export cookie 1297408554816629651 [ 9393.930312] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.935691] LustreError: 218853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9394.349526] Lustre: server umount lustre-MDT0001 complete [ 9404.410267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.061756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.077279] Lustre: Skipped 5 previous similar messages [ 9411.690448] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9425.053545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9432.032542] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9443.105690] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9450.911698] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9459.849862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.263638] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9475.740676] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9489.903637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9489.913870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.924288] Lustre: Skipped 2 previous similar messages [ 9493.270387] Lustre: server umount lustre-OST0000 complete [ 9503.699914] Lustre: server umount lustre-MDT0000 complete [ 9508.635226] LustreError: 223029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841026 with bad export cookie 1297408554816630526 [ 9508.661265] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9509.364141] Lustre: server umount lustre-MDT0001 complete [ 9615.967954] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9619.546545] Key type lgssc unregistered [ 9619.880906] LNet: 226713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9619.896288] LNetError: 226713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9619.937186] LNet: Removed LNI 192.168.201.127@tcp [ 9620.988526] Key type .llcrypt unregistered [ 9620.990152] Key type ._llcrypt unregistered [ 9651.688743] Key type ._llcrypt registered [ 9651.689976] Key type .llcrypt registered [ 9651.920872] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9668.857580] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9670.234370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9670.526100] alg: No test for adler32 (adler32-zlib) [ 9671.983365] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 9672.255281] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9674.003094] Key type lgssc registered [ 9675.720569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9717.980268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9718.038188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.279881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9719.351633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9719.472124] Lustre: lustre-MDT0000: new disk, initializing [ 9719.596612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.673160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9725.405459] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9738.786391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.972266] Lustre: 230077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9739.143230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9739.157253] Lustre: Skipped 1 previous similar message [ 9739.422351] Lustre: lustre-MDT0001: new disk, initializing [ 9739.629727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.712237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9739.726915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9746.171472] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9759.125430] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9766.933306] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9775.688217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.153177] Lustre: lustre-OST0000: new disk, initializing [ 9776.158317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9776.165522] Lustre: 232046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9776.283486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.946619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9783.961535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9784.086765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9784.847487] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9795.689540] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9801.226891] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9801.418467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9806.763466] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9807.052282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9809.894856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9809.905511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.916336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9809.926507] Lustre: Skipped 1 previous similar message [ 9809.951628] Lustre: Skipped 1 previous similar message [ 9815.011614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9815.785798] Lustre: server umount lustre-OST0000 complete [ 9821.667291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9821.677091] LustreError: Skipped 1 previous similar message [ 9821.685444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9821.728448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.737139] Lustre: Skipped 1 previous similar message [ 9825.251570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.252577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9825.299457] Lustre: Skipped 1 previous similar message [ 9826.707956] Lustre: server umount lustre-MDT0000 complete [ 9831.738837] LustreError: 230069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841349 with bad export cookie 12143876278867673627 [ 9831.741595] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9831.750341] LustreError: 230069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9832.048143] Lustre: server umount lustre-MDT0001 complete [ 9855.032391] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9867.830582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.227186] LustreError: 234980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.258403] LustreError: 234980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9868.314252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9873.377973] LustreError: 234981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.462287] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9878.508326] LustreError: 234980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.619042] LustreError: 234981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.403444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.250158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9893.349631] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9897.075874] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9904.226159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9904.541986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.431959] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9920.459114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.844476] Lustre: lustre-OST0001: new disk, initializing [ 9920.861031] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9920.870899] Lustre: 237287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9921.080707] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9927.848837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9927.856194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9928.112021] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9929.270833] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9943.649305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9950.438521] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9966.430605] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:24:42 (1788841482) [ 9974.243628] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9974.270413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9974.281080] Lustre: Skipped 2 previous similar messages [ 9974.284967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9974.370244] Lustre: Skipped 3 previous similar messages [ 9976.922179] Lustre: server umount lustre-MDT0000 complete [ 9979.362363] LustreError: 234975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.398461] LustreError: 234975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9982.302485] LustreError: 237286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841500 with bad export cookie 12143876278867674509 [ 9982.334914] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9982.869246] Lustre: server umount lustre-MDT0001 complete [ 9997.103275] Lustre: server umount lustre-OST0000 complete [10000.607226] Lustre: 228297:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841502/real 1788841502] req@ffff95cb41e0d880 x1875735939767296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10000.659378] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.665516] Lustre: Skipped 1 previous similar message [10002.390087] Lustre: server umount lustre-OST0001 complete [10008.702040] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [10020.446803] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10063.613423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10063.987703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10064.023905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10064.163611] Lustre: lustre-MDT0000: new disk, initializing [10064.367421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10064.407495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10070.933621] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10084.028614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.216600] Lustre: 241239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10084.366898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10084.372271] Lustre: Skipped 1 previous similar message [10084.527893] Lustre: lustre-MDT0001: new disk, initializing [10084.700230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10084.712039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10090.854535] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10102.901802] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10109.807897] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10116.945760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.330637] Lustre: lustre-OST0000: new disk, initializing [10117.339384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10117.353875] Lustre: 243210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10118.647082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10118.674847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10118.866179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10125.343841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10137.654640] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10143.287019] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10143.513790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10149.293697] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10149.506906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10154.982049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10154.983062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.986810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10154.993015] LustreError: Skipped 1 previous similar message [10155.013462] Lustre: Skipped 1 previous similar message [10166.241882] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10166.647693] Lustre: server umount lustre-OST0000 complete [10171.877063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10171.891496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10171.914489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.930226] Lustre: Skipped 5 previous similar messages [10177.399553] Lustre: server umount lustre-MDT0000 complete [10180.588423] LustreError: 241251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.616165] LustreError: 241251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10181.987577] LustreError: 241231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841700 with bad export cookie 12143876278867676644 [10181.995145] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.997298] LustreError: 241231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10182.539369] Lustre: server umount lustre-MDT0001 complete [10193.457274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.258126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10194.262616] Lustre: Skipped 2 previous similar messages [10199.524509] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.550665] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10200.424493] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10211.031300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.278211] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10227.292533] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.664499] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10243.058692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10250.726257] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10259.942967] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10405.345347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10405.350863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10405.360837] Lustre: Skipped 1 previous similar message [10405.392418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10405.408708] Lustre: Skipped 2 previous similar messages [10411.172725] Lustre: server umount lustre-OST0000 complete [10422.487529] Lustre: server umount lustre-MDT0000 complete [10422.761270] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.788181] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10427.602143] LustreError: 245410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841945 with bad export cookie 12143876278867677526 [10427.606967] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10427.612083] LustreError: 245410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10428.076468] Lustre: server umount lustre-MDT0001 complete [10529.714906] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10534.005691] Key type lgssc unregistered [10534.404252] LNet: 249038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10534.425960] LNetError: 249038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10534.452566] LNet: Removed LNI 192.168.201.127@tcp [10535.407261] Key type .llcrypt unregistered [10535.411133] Key type ._llcrypt unregistered [10567.686734] Key type ._llcrypt registered [10567.695355] Key type .llcrypt registered [10567.887474] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [10584.758513] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10586.213705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10586.254990] alg: No test for adler32 (adler32-zlib) [10587.517552] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [10587.858113] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10589.671223] Key type lgssc registered [10591.192829] Lustre: Echo OBD driver; http://www.lustre.org/ [10645.489941] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10660.058385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10660.082479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.477105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10661.535027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10661.669912] Lustre: lustre-MDT0000: new disk, initializing [10661.838245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10661.865135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10666.429699] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10676.237225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.371672] Lustre: 253108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10676.434283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10676.438990] Lustre: Skipped 1 previous similar message [10676.552630] Lustre: lustre-MDT0001: new disk, initializing [10676.657593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10676.694084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10676.700992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10682.353471] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10688.926295] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10697.365510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10697.832915] Lustre: lustre-OST0000: new disk, initializing [10697.844447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10697.855844] Lustre: 254739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10698.042722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10702.933323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10702.943025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10703.082654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10705.422596] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10717.948231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10718.155508] Lustre: lustre-OST0001: new disk, initializing [10718.160678] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10718.170885] Lustre: 255612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10718.299574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10723.916906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10723.934768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10724.028267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10725.065517] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10735.653849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10742.628936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10753.505059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10753.517595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10753.552845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.049276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.049897] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.074322] Lustre: Skipped 1 previous similar message [10760.167872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10760.170930] Lustre: Skipped 3 previous similar messages [10765.302625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10765.325034] Lustre: Skipped 4 previous similar messages [10767.327480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10767.607490] Lustre: server umount lustre-MDT0000 complete [10771.354359] LustreError: 253101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842289 with bad export cookie 18052665206741290452 [10771.356806] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.373506] LustreError: 253101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10771.830291] Lustre: server umount lustre-MDT0001 complete [10787.253519] Lustre: server umount lustre-OST0000 complete [10801.802698] Lustre: server umount lustre-OST0001 complete [10819.478304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10832.411910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10843.110128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10855.994167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10881.406743] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10894.824683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.846269] Lustre: lustre-MDT0000: reset Object Index mappings [10895.307257] LustreError: 260625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.336438] LustreError: 260625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10895.403934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10900.454693] LustreError: 260626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.478374] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10905.573380] LustreError: 260625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.695138] LustreError: 260626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.223427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.276728] Lustre: lustre-MDT0001: reset Object Index mappings [10912.865716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10918.436921] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10921.777799] Lustre: 261768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10930.297425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10930.737737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10936.883482] LustreError: 262124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.743399] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10939.982071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10945.014486] LustreError: 262434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.041426] LustreError: 262434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10947.989043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10953.733890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10956.100786] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10965.191675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10969.469288] Lustre: 263640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10979.303076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10979.308652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10979.320990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10979.345661] Lustre: Skipped 3 previous similar messages [10979.360060] Lustre: Skipped 3 previous similar messages [10982.391579] Lustre: server umount lustre-MDT0000 complete [10984.433026] LustreError: 260622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.448371] LustreError: 260622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10986.632568] LustreError: 263643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842504 with bad export cookie 18052665206741297242 [10986.636974] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.647838] LustreError: 263643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10987.502770] Lustre: server umount lustre-MDT0001 complete [11002.923308] Lustre: server umount lustre-OST0000 complete [11004.909441] Lustre: 250609:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842507/real 1788842507] req@ffff95cc7a0df800 x1875736900272896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788842523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11004.947205] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.954582] Lustre: Skipped 1 previous similar message [11006.943193] Lustre: 250612:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842509/real 1788842509] req@ffff95cc7a0dd180 x1875736900273152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788842525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11007.692769] Lustre: server umount lustre-OST0001 complete [11029.845469] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:42:26 (1788842546) [11049.325691] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [11061.871331] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11107.405296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.667862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11107.694993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11107.838963] Lustre: lustre-MDT0000: new disk, initializing [11107.946281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11107.957960] Lustre: Skipped 1 previous similar message [11107.984674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11112.994147] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11125.571767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.685680] Lustre: 267899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11125.752817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11125.757215] Lustre: Skipped 1 previous similar message [11125.823233] Lustre: lustre-MDT0001: new disk, initializing [11125.939826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11125.958154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11131.591735] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11144.329523] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11152.437976] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11161.050172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11161.343831] Lustre: lustre-OST0000: new disk, initializing [11161.347181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11161.352974] Lustre: 269865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11163.286096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11163.291098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11163.416617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11168.432401] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11179.093886] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11184.831824] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11185.190940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11190.903628] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11191.293833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11194.366849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.372295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11194.394827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11194.407108] LustreError: Skipped 1 previous similar message [11194.417448] Lustre: Skipped 1 previous similar message [11199.807787] Lustre: server umount lustre-OST0000 complete [11204.583565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.590634] Lustre: Skipped 2 previous similar messages [11214.818944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11214.826553] Lustre: Skipped 6 previous similar messages [11218.399172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11218.687841] Lustre: server umount lustre-MDT0000 complete [11219.940497] LustreError: 267910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.966104] LustreError: 267910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11222.653510] LustreError: 267891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842740 with bad export cookie 18052665206741299335 [11222.657962] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11222.660360] LustreError: 267891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11223.101938] Lustre: server umount lustre-MDT0001 complete [11242.020333] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11253.040902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.280697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11253.284877] Lustre: Skipped 2 previous similar messages [11260.886738] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11271.837832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.499076] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11282.207270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11291.093795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11299.290657] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11310.405744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11310.683480] Lustre: lustre-OST0001: new disk, initializing [11310.686597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11310.694936] Lustre: 275117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11312.008884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11312.018964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11312.207558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11317.919643] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11329.061372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11334.577414] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11341.574719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11341.665058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11353.057954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.079982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.086475] Lustre: Skipped 2 previous similar messages [11353.108519] Lustre: Skipped 3 previous similar messages [11356.303598] Lustre: server umount lustre-MDT0000 complete [11358.194192] LustreError: 272802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.227690] LustreError: 272802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11360.584749] LustreError: 272787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842878 with bad export cookie 18052665206741300217 [11360.588604] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11360.601510] LustreError: 272787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11360.948809] Lustre: server umount lustre-MDT0001 complete [11375.856079] Lustre: server umount lustre-OST0000 complete [11379.167206] Lustre: 250611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842881/real 1788842881] req@ffff95cb47cd2300 x1875736900389888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788842897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11380.706502] Lustre: server umount lustre-OST0001 complete [11401.142744] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11413.690117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11414.305581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.316708] Lustre: Skipped 3 previous similar messages [11419.431839] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11429.334584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.879476] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11438.343918] Lustre: 279315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11445.920449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11453.532484] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11459.708107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11463.308615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11468.777646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11470.766757] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11480.307455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11484.609657] Lustre: 281193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11491.351554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11491.446205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11506.142538] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:50:22 (1788843022) [11525.090276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11525.100065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11525.130095] Lustre: Skipped 4 previous similar messages [11525.141524] Lustre: Skipped 3 previous similar messages [11528.978596] Lustre: server umount lustre-MDT0000 complete [11530.221484] LustreError: 278175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.262472] LustreError: 278175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11534.623927] LustreError: 278156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843052 with bad export cookie 18052665206741306769 [11534.653072] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11541.695623] Lustre: server umount lustre-MDT0001 complete [11553.479595] Lustre: server umount lustre-OST0000 complete [11558.351532] Lustre: server umount lustre-OST0001 complete [11566.998057] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [11579.604972] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11636.747276] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11648.817520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.055310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11649.104488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11649.229804] Lustre: lustre-MDT0000: new disk, initializing [11649.328623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11655.583108] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11667.078380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.136473] Lustre: 285438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11667.317085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11671.616722] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11690.148789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11690.573222] Lustre: 287071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11690.710214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11690.717918] Lustre: Skipped 5 previous similar messages [11693.017458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11699.164441] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11711.797344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.025518] Lustre: 287942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11713.948662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11713.953720] Lustre: Skipped 1 previous similar message [11714.075720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11719.711721] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11730.109737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11735.307344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11735.314320] Lustre: Skipped 1 previous similar message [11754.120869] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:54:30 (1788843270) [11759.585520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11759.592274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.617702] Lustre: Skipped 3 previous similar messages [11759.626844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11759.635121] Lustre: Skipped 4 previous similar messages [11763.844615] Lustre: server umount lustre-MDT0000 complete [11768.031114] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843286 with bad export cookie 18052665206741314420 [11768.038614] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11768.055879] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11768.445188] Lustre: server umount lustre-MDT0001 complete [11783.290742] Lustre: server umount lustre-OST0000 complete [11786.207206] Lustre: 250611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843288/real 1788843288] req@ffff95cb47cd3800 x1875736900550656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11788.161564] Lustre: server umount lustre-OST0001 complete [11800.600748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.670492] LustreError: 290191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.716907] LustreError: 290191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11807.890422] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11819.193447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.619949] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11832.617247] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11839.350895] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11847.637277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11854.572537] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11862.696601] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11894.562586] loop1: detected capacity change from 0 to 204800000 [11894.853270] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11894.878881] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11894.906326] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11901.195730] loop1: detected capacity change from 0 to 204800000 [11907.985368] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.097175] LustreError: 290180:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25163: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11908.128386] LustreError: 290180:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25163 [11908.138047] LustreError: 290180:0:(mgs_handler.c:517:mgs_target_reg()) fs25163-MDT0001: Error checking target index: rc = -13 [11908.149915] LustreError: fs25163-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11908.157603] LustreError: 293929:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11908.166691] LustreError: 293929:0:(tgt_mount.c:2132:server_put_super()) no obd fs25163-MDT0001 [11908.170811] LustreError: 293929:0:(tgt_mount.c:116:server_deregister_mount()) fs25163-MDT0001 not registered [11908.199437] Lustre: server umount fs25163-MDT0001 complete [11908.203791] LustreError: 293929:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11918.610466] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.700540] Lustre: Setting parameter fs25163-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25163-MDT0001 [11918.754191] Lustre: srv-fs25163-MDT0001: No data found on store. Initialize space. [11918.764786] Lustre: Skipped 4 previous similar messages [11918.894647] Lustre: fs25163-MDT0001: new disk, initializing [11918.898219] Lustre: Skipped 3 previous similar messages [11919.074430] Lustre: srv-fs25163-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11919.085860] Lustre: 294397:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25163-MDT0001-mdtlov: can't create new llog: rc = -115 [11919.101395] LustreError: 294397:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25163-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11925.028875] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11933.333645] Lustre: server umount fs25163-MDT0001 complete [11943.406892] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11948.998718] loop1: detected capacity change from 0 to 204800000 [11955.373090] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.915360] Lustre: 295967:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25163-MDT0001-mdtlov: can't create new llog: rc = -115 [11955.929572] LustreError: 295967:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25163-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11961.319365] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11967.889148] Lustre: server umount fs25163-MDT0001 complete [11983.841118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11983.846154] LustreError: Skipped 1 previous similar message [11983.848795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.857287] Lustre: Skipped 4 previous similar messages [11985.987558] Lustre: server umount lustre-MDT0000 complete [11995.979897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.102313] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12001.041362] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12001.788562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12010.454516] loop1: detected capacity change from 0 to 204800000 [12018.037492] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.692337] Lustre: 298415:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25163-MDT0001-mdtlov: can't create new llog: rc = -115 [12018.707110] LustreError: 298415:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25163-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12023.790985] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12029.147270] Lustre: server umount fs25163-MDT0001 complete [12058.091218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12058.104222] Lustre: Skipped 10 previous similar messages [12069.343644] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12069.594273] Lustre: server umount lustre-OST0000 complete [12073.445027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12079.498446] Lustre: server umount lustre-MDT0000 complete [12084.048645] LustreError: 290173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843602 with bad export cookie 18052665206741317101 [12084.064196] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.076476] LustreError: 290173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12084.995970] Lustre: server umount lustre-MDT0001 complete [12103.415885] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12106.928145] Key type lgssc unregistered [12107.352315] LNet: 300780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12107.364813] LNetError: 300780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12107.382947] LNet: Removed LNI 192.168.201.127@tcp [12108.771083] Key type .llcrypt unregistered [12108.778067] Key type ._llcrypt unregistered [12128.371317] Key type ._llcrypt registered [12128.375991] Key type .llcrypt registered [12128.528920] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12129.933382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12129.971495] alg: No test for adler32 (adler32-zlib) [12131.067656] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [12131.405185] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12133.111224] Key type lgssc registered [12134.083735] Lustre: Echo OBD driver; http://www.lustre.org/ [12141.485515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12141.502797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12143.242049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.753508] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12152.808238] LustreError: 302048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.825143] LustreError: 302048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12157.355860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12158.044189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12162.572737] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12171.569111] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12177.980773] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.614394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12186.944258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12187.954968] LustreError: 303889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.215115] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12193.264281] LustreError: 303889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.380648] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.564375] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12203.496209] LustreError: 303889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.545056] LustreError: 303889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12213.746906] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.760994] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12222.713149] loop1: detected capacity change from 0 to 204800000 [12222.971144] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12222.977910] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12222.997704] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12230.108860] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.141546] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [12230.478469] loop1: detected capacity change from 0 to 204800000 [12237.526544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12237.858829] Lustre: fs3425-OST0000: new disk, initializing [12237.864860] Lustre: srv-fs3425-OST0000: No data found on store. Initialize space. [12237.878722] Lustre: 305799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12238.034696] Lustre: fs3425-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12246.697148] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12256.018840] Lustre: server umount fs3425-OST0000 complete [12264.943951] LustreError: 303890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.960095] LustreError: 303890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [12268.672301] loop1: detected capacity change from 0 to 204800000 [12270.057835] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12275.186792] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12275.341952] Lustre: fs3425-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12281.981886] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12285.435731] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12291.451697] loop2: detected capacity change from 0 to 204800000 [12291.719672] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12291.741409] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12291.752950] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12297.722732] loop2: detected capacity change from 0 to 204800000 [12303.725308] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12303.825449] LustreError: fs3425-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12303.833973] LustreError: fs3425-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12303.840883] LustreError: 308843:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [12303.844840] LustreError: 308843:0:(tgt_mount.c:2132:server_put_super()) no obd fs3425-OST0001 [12303.848476] LustreError: 308843:0:(tgt_mount.c:116:server_deregister_mount()) fs3425-OST0001 not registered [12303.876810] Lustre: server umount fs3425-OST0001 complete [12303.879294] LustreError: 308843:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12315.275949] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12315.382455] Lustre: fs3425-OST0001: new disk, initializing [12315.386275] Lustre: srv-fs3425-OST0001: No data found on store. Initialize space. [12315.394575] Lustre: 309394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12315.526080] Lustre: fs3425-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12322.777357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12331.492912] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12331.522402] LustreError: 303892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12332.152190] Lustre: server umount fs3425-OST0000 complete [12343.053945] Lustre: server umount fs3425-OST0001 complete [12356.621481] loop1: detected capacity change from 0 to 204800000 [12362.989286] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.314457] Lustre: fs3425-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12370.682622] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12377.432720] loop2: detected capacity change from 0 to 204800000 [12383.137181] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12391.886417] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12403.512716] Lustre: server umount fs3425-OST0000 complete [12414.891247] Lustre: server umount fs3425-OST0001 complete [12423.679625] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12433.892785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12433.902563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12433.921291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12433.946938] Lustre: Skipped 1 previous similar message [12439.028337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.032962] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12439.040473] Lustre: Skipped 2 previous similar messages [12444.130567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12444.137184] Lustre: Skipped 1 previous similar message [12445.151649] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12445.350246] Lustre: server umount lustre-OST0000 complete [12449.760657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12449.772160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12449.792440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.368249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.368729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.406417] Lustre: Skipped 1 previous similar message [12456.286462] Lustre: server umount lustre-MDT0000 complete [12459.496060] LustreError: 302049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.518251] LustreError: 302049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 64 previous similar messages [12459.833722] LustreError: 302028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843978 with bad export cookie 1409738032522957797 [12459.835038] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12459.853062] LustreError: 302028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12460.198276] Lustre: server umount lustre-MDT0001 complete [12471.546876] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12474.356507] Key type lgssc unregistered [12474.632701] LNet: 315318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12474.637297] LNetError: 315318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12474.648852] LNet: Removed LNI 192.168.201.127@tcp [12475.642376] Key type .llcrypt unregistered [12475.644751] Key type ._llcrypt unregistered [12516.156566] Key type ._llcrypt registered [12516.159682] Key type .llcrypt registered [12516.320258] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [12534.860362] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12536.547373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12536.587183] alg: No test for adler32 (adler32-zlib) [12537.724798] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [12537.993083] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12539.687275] Key type lgssc registered [12541.473221] Lustre: Echo OBD driver; http://www.lustre.org/ [12582.608377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12582.645942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.898976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12583.927133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12584.009778] Lustre: lustre-MDT0000: new disk, initializing [12584.073405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12584.097617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12589.552908] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12601.572964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.726928] Lustre: 319179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12601.773499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12601.777155] Lustre: Skipped 1 previous similar message [12601.840820] Lustre: lustre-MDT0001: new disk, initializing [12601.910473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.944250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12601.960409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12607.305228] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12617.659471] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12626.009127] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12635.569188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12635.993075] Lustre: lustre-OST0000: new disk, initializing [12636.002804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12636.017815] Lustre: 321148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12636.160610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12644.747408] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12645.973072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12645.977234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12646.078934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12656.034652] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12658.195733] Lustre: Mounted lustre-client - version 2.17.57_110_g063755c [12659.194142] Lustre: Unmounted lustre-client [12671.980287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12671.980721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.989364] LustreError: Skipped 1 previous similar message [12671.995128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12672.007918] Lustre: Skipped 1 previous similar message [12672.021818] Lustre: Skipped 1 previous similar message [12673.510555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12676.703281] Lustre: server umount lustre-OST0000 complete [12683.749565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12683.750048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12683.795152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12683.805929] Lustre: Skipped 2 previous similar messages [12688.966859] Lustre: server umount lustre-MDT0000 complete [12692.789965] LustreError: 319173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844211 with bad export cookie 13209189595036057499 [12692.801437] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12693.153549] Lustre: server umount lustre-MDT0001 complete [12705.881841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12708.481980] Key type lgssc unregistered [12719.266708] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [12719.970423] LNet: 323314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12719.995462] LNetError: 323314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12789.679677] LNet: Removed LNI 192.168.201.127@tcp [12790.710731] Key type .llcrypt unregistered [12790.711821] Key type ._llcrypt unregistered [12810.797395] Key type ._llcrypt registered [12810.805741] Key type .llcrypt registered [12810.962941] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [12829.430632] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12830.249476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12830.283105] alg: No test for adler32 (adler32-zlib) [12831.822240] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [12832.069128] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12833.831454] Key type lgssc registered [12835.202578] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.744685] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:13:08 (1788844388) [12879.547599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12879.588262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.914285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12880.966930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12881.062478] Lustre: lustre-MDT0000: new disk, initializing [12881.133981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12881.151134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12886.442298] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12897.993207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.180291] Lustre: 326533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12898.237281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12898.241856] Lustre: Skipped 1 previous similar message [12898.331375] Lustre: lustre-MDT0001: new disk, initializing [12898.418262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12898.461832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12898.481345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12903.754242] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12915.260803] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12921.605166] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12928.507812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12928.814215] Lustre: lustre-OST0000: new disk, initializing [12928.819990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12928.825749] Lustre: 328503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12928.905996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12936.470627] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12938.809682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12938.818843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12938.945190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12947.097681] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12964.836351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12964.846086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12964.881533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12964.884312] Lustre: Skipped 1 previous similar message [12964.898887] Lustre: Skipped 1 previous similar message [12969.958450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12969.970035] Lustre: Skipped 1 previous similar message [12970.381696] Lustre: server umount lustre-OST0000 complete [12975.585924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12975.589056] LustreError: Skipped 1 previous similar message [12975.593533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12975.626788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12980.200683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12980.201182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12980.252530] Lustre: Skipped 1 previous similar message [12981.752896] Lustre: server umount lustre-MDT0000 complete [12985.619171] LustreError: 326525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844503 with bad export cookie 15563880596658154983 [12985.620346] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12985.636163] LustreError: 326525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12986.104305] Lustre: server umount lustre-MDT0001 complete [12997.454284] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13000.404816] Key type lgssc unregistered [13000.800746] LNet: 330410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13000.807399] LNetError: 330410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13000.831437] LNet: Removed LNI 192.168.201.127@tcp [13001.771708] Key type .llcrypt unregistered [13001.774109] Key type ._llcrypt unregistered [13025.924239] Key type ._llcrypt registered [13025.929127] Key type .llcrypt registered [13025.993986] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13026.782927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13026.890779] alg: No test for adler32 (adler32-zlib) [13028.444669] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [13028.839409] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13030.527885] Key type lgssc registered [13032.180740] Lustre: Echo OBD driver; http://www.lustre.org/ [13041.230970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13041.251347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13042.740620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13048.594566] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13051.886616] LustreError: 331687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.919017] LustreError: 331687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13056.995610] LustreError: 331686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.336768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13059.809944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13065.475259] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13074.864245] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13082.711732] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13092.677355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13093.051503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13101.459397] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13110.638522] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13125.348746] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13127.203477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13131.756753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13131.757334] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [13131.775131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13131.797969] Lustre: Skipped 1 previous similar message [13133.280341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13133.285283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13133.324354] Lustre: Skipped 1 previous similar message [13136.839907] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [13136.857172] Lustre: Skipped 2 previous similar messages [13137.881970] Lustre: server umount lustre-MDT0000 complete [13138.404610] LustreError: 331687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13138.413812] LustreError: 331687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13141.970840] LustreError: 334181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.615237] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844660 with bad export cookie 4416353786221644206 [13142.626885] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13143.525736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13143.542078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13147.078146] LustreError: 331683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13147.104581] LustreError: 331683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13147.556046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13147.564329] Lustre: Skipped 1 previous similar message [13149.277729] Lustre: server umount lustre-MDT0001 complete [13153.737185] Lustre: server umount lustre-OST0000 complete [13163.454491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.006642] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13172.152790] Lustre: 335872:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.127@tcp [13175.017576] Lustre: server umount MGS complete [13185.376954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13185.940770] LustreError: 336445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.956323] LustreError: 336445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13186.094524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13191.762734] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13201.915197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13202.279733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13207.704907] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13217.148284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13217.596046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13225.616763] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13235.245561] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13237.137533] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13243.403371] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13245.537936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13251.553383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13251.565721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13251.571800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13251.595210] Lustre: Skipped 1 previous similar message [13252.758083] Lustre: server umount lustre-OST0000 complete [13258.720627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13258.730768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13263.681804] Lustre: server umount lustre-MDT0000 complete [13266.916797] LustreError: 336441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13266.933896] LustreError: 336441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [13267.942702] LustreError: 336426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844786 with bad export cookie 4416353786221645053 [13267.948601] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13267.956648] LustreError: 336426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13268.282863] Lustre: server umount lustre-MDT0001 complete [13281.664208] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13284.692974] Key type lgssc unregistered [13285.048795] LNet: 339879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13285.059443] LNetError: 339879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13285.082278] LNet: Removed LNI 192.168.201.127@tcp [13286.209810] Key type .llcrypt unregistered [13286.212069] Key type ._llcrypt unregistered [13305.772784] Key type ._llcrypt registered [13305.779238] Key type .llcrypt registered [13305.941394] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [13323.654121] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13325.377274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13325.388626] alg: No test for adler32 (adler32-zlib) [13326.616506] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [13326.917508] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13328.607963] Key type lgssc registered [13330.175980] Lustre: Echo OBD driver; http://www.lustre.org/ [13371.335112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13371.374469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13372.582331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13372.605131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13372.674197] Lustre: lustre-MDT0000: new disk, initializing [13372.736473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13372.768391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13377.577786] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13388.375808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13388.488302] Lustre: 343038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13388.566699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13388.574083] Lustre: Skipped 1 previous similar message [13388.704370] Lustre: lustre-MDT0001: new disk, initializing [13388.811687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13388.860369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13388.880427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13392.851807] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13402.613681] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13408.469759] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13417.537219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13417.923153] Lustre: lustre-OST0000: new disk, initializing [13417.934296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13417.946241] Lustre: 345010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13418.202580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13425.924500] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13427.395196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13427.404383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13427.585200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13436.164342] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13440.981108] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13441.185755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13446.407803] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13446.690889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13453.282848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13453.286337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13453.298076] LustreError: Skipped 1 previous similar message [13453.303242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13453.304718] Lustre: Skipped 1 previous similar message [13453.324282] Lustre: Skipped 1 previous similar message [13455.414308] Lustre: server umount lustre-OST0000 complete [13460.463170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13460.473956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13460.498524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13463.521161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13463.521982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.979747] Lustre: server umount lustre-MDT0000 complete [13470.134151] LustreError: 343032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844988 with bad export cookie 4969707168022934477 [13470.138628] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13470.143120] LustreError: 343032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13470.459714] Lustre: server umount lustre-MDT0001 complete [13480.209137] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:23:16 (1788844996) [13499.249672] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [13509.593040] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13551.274463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13551.524084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13551.557988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13551.666899] Lustre: lustre-MDT0000: new disk, initializing [13551.772846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13551.805786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13557.886176] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13570.162308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13570.375362] Lustre: 349769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13570.497744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13570.505946] Lustre: Skipped 1 previous similar message [13570.653796] Lustre: lustre-MDT0001: new disk, initializing [13570.830039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13570.895197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13570.907189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13576.742952] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13587.531434] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13594.011957] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13601.092794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13601.253640] Lustre: lustre-OST0000: new disk, initializing [13601.255905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13601.259271] Lustre: 351742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13601.328889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13609.235258] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13610.674877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13610.699498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13610.914784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13622.359719] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13662.178975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13662.182973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13662.188125] LustreError: Skipped 1 previous similar message [13662.192106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13662.198909] Lustre: Skipped 1 previous similar message [13662.217993] Lustre: Skipped 2 previous similar messages [13667.298742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13667.302724] Lustre: Skipped 1 previous similar message [13667.857561] Lustre: server umount lustre-OST0000 complete [13672.426019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13672.458883] Lustre: Skipped 1 previous similar message [13674.560969] Lustre: server umount lustre-MDT0000 complete [13677.541901] LustreError: 349782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13677.557523] LustreError: 349782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13679.029188] LustreError: 349763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845197 with bad export cookie 4969707168022935359 [13679.037108] LustreError: 349763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13679.038399] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13679.495290] Lustre: server umount lustre-MDT0001 complete [13692.647839] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13695.468780] Key type lgssc unregistered [13695.766581] LNet: 353916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13695.772631] LNetError: 353916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13695.799884] LNet: Removed LNI 192.168.201.127@tcp [13696.681789] Key type .llcrypt unregistered [13696.684321] Key type ._llcrypt unregistered [13798.247590] Key type ._llcrypt registered [13798.250820] Key type .llcrypt registered [13798.400066] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13798.752904] Key type .llcrypt unregistered [13798.754644] Key type ._llcrypt unregistered [13885.335365] Key type ._llcrypt registered [13885.338332] Key type .llcrypt registered [13885.429815] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13885.594290] Key type .llcrypt unregistered [13885.596248] Key type ._llcrypt unregistered [13921.870686] Key type ._llcrypt registered [13921.876550] Key type .llcrypt registered [13922.114411] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13922.411562] Key type .llcrypt unregistered [13922.414163] Key type ._llcrypt unregistered [13956.422286] Key type ._llcrypt registered [13956.424947] Key type .llcrypt registered [13956.530823] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13956.786387] Key type .llcrypt unregistered [13956.789201] Key type ._llcrypt unregistered [14046.457944] Key type ._llcrypt registered [14046.466164] Key type .llcrypt registered [14046.603225] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14046.924344] Key type .llcrypt unregistered [14046.928252] Key type ._llcrypt unregistered [14082.367840] Key type ._llcrypt registered [14082.371733] Key type .llcrypt registered [14082.535882] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14082.840340] Key type .llcrypt unregistered [14082.842475] Key type ._llcrypt unregistered [14120.497281] Key type ._llcrypt registered [14120.499847] Key type .llcrypt registered [14120.633150] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14120.990331] Key type .llcrypt unregistered [14120.993866] Key type ._llcrypt unregistered [14154.553622] Key type ._llcrypt registered [14154.556212] Key type .llcrypt registered [14154.729632] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14155.099311] Key type .llcrypt unregistered [14155.102565] Key type ._llcrypt unregistered [14191.183783] Key type ._llcrypt registered [14191.186196] Key type .llcrypt registered [14191.296637] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14191.572787] Key type .llcrypt unregistered [14191.575786] Key type ._llcrypt unregistered [14224.678283] Key type ._llcrypt registered [14224.680268] Key type .llcrypt registered [14224.775827] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14225.104778] Key type .llcrypt unregistered [14225.111494] Key type ._llcrypt unregistered [14302.265963] Key type ._llcrypt registered [14302.270957] Key type .llcrypt registered [14302.435904] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14302.806372] Key type .llcrypt unregistered [14302.818149] Key type ._llcrypt unregistered [14334.866434] Key type ._llcrypt registered [14334.867460] Key type .llcrypt registered [14334.932357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14335.109495] Key type .llcrypt unregistered [14335.113937] Key type ._llcrypt unregistered [14371.309934] Key type ._llcrypt registered [14371.315088] Key type .llcrypt registered [14371.475660] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14371.760602] Key type .llcrypt unregistered [14371.763131] Key type ._llcrypt unregistered [14431.225487] Key type ._llcrypt registered [14431.227158] Key type .llcrypt registered [14431.306804] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14450.394099] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14452.202573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14452.239152] alg: No test for adler32 (adler32-zlib) [14453.551335] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [14453.919906] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [14455.679319] Key type lgssc registered [14457.383535] Lustre: Echo OBD driver; http://www.lustre.org/ [14501.417090] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14501.440829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14502.892233] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14502.936740] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14503.078702] Lustre: tst250fs-MDT0000: new disk, initializing [14503.168477] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14503.201927] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14508.393273] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14519.686535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14519.872935] Lustre: 374200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14519.912987] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14519.916843] Lustre: Skipped 1 previous similar message [14520.029332] Lustre: tst250fs-MDT0001: new disk, initializing [14520.174514] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14520.228305] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14520.247648] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14525.467140] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14531.028178] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14539.188262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14539.615540] Lustre: tst250fs-OST0000: new disk, initializing [14539.623560] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14539.632271] Lustre: 375834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14539.763617] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14543.742474] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14543.767100] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14543.952151] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14548.228815] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14560.249087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14560.449479] Lustre: tst250fs-OST0001: new disk, initializing [14560.455595] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14560.460931] Lustre: 376705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14560.580714] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14565.779590] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14567.145859] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14567.154336] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14567.361049] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14571.257460] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14573.585244] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14573.594222] Lustre: Skipped 1 previous similar message [14597.088612] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14597.108944] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14597.138376] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14598.136126] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14598.146862] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14598.153447] Lustre: Skipped 1 previous similar message [14598.183919] Lustre: Skipped 2 previous similar messages [14603.234985] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14603.258411] Lustre: Skipped 3 previous similar messages [14608.354651] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14608.367275] Lustre: Skipped 3 previous similar messages [14609.887288] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14610.119421] Lustre: server umount tst250fs-MDT0000 complete [14614.344764] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846132 with bad export cookie 4243930240768053085 [14614.351304] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14614.359335] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14614.642911] Lustre: server umount tst250fs-MDT0001 complete [14629.729372] Lustre: server umount tst250fs-OST0000 complete [14644.312850] Lustre: server umount tst250fs-OST0001 complete [14665.044724] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14674.943263] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14711.333240] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:43:48 (1788846228) [14713.327751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14733.328084] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14750.949898] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14792.889924] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14516 sec ========= 01:45:09 (1788846309) [14795.832713] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:45:11 (1788846311) === [14801.695524] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:45:17 (1788846317) === [14855.537576] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14859.835984] Key type lgssc unregistered [14860.271258] LNet: 385875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14860.286042] LNetError: 385875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14860.317017] LNet: Removed LNI 192.168.201.127@tcp [14861.348343] Key type .llcrypt unregistered [14861.352288] Key type ._llcrypt unregistered