[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500024560 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004025] kvm-guest: setup PV IPIs [ 0.008520] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011176] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.015085] *** VALIDATE selinux *** [ 0.024338] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029584] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030195] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033074] *** VALIDATE tmpfs *** [ 0.034407] *** VALIDATE proc *** [ 0.035207] *** VALIDATE cgroup *** [ 0.036005] *** VALIDATE cgroup2 *** [ 0.037284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040022] Spectre V2 : User space: Vulnerable [ 0.041004] Speculative Store Bypass: Vulnerable [ 0.044575] debug: unmapping init [mem 0xffffffff9fa59000-0xffffffff9fa60fff] [ 0.046273] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047840] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050000] ... generic registers: 4 [ 0.051009] ... value mask: 0000ffffffffffff [ 0.052009] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.053961] ... event mask: 000000070000000f [ 0.054328] rcu: Hierarchical SRCU implementation. [ 0.056514] smp: Bringing up secondary CPUs ... [ 0.057537] x86: Booting SMP configuration: [ 0.058023] .... node #0, CPUs: #1 #2 #3 [ 0.065214] smp: Brought up 1 node, 4 CPUs [ 0.067023] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105214] node 0 deferred pages initialised in 35ms [ 0.112295] devtmpfs: initialized [ 0.114591] x86/mm: Memory block size: 128MB [ 0.118823] gcov: version magic: 0x41383552 [ 0.122390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.124074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125342] pinctrl core: initialized pinctrl subsystem [ 0.127200] [ 0.127893] ************************************************************* [ 0.130015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133008] ** ** [ 0.135011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138015] ** ** [ 0.141014] ** This means that this kernel is built to expose internal ** [ 0.144014] ** IOMMU data structures, which may compromise security on ** [ 0.147012] ** your system. ** [ 0.149011] ** ** [ 0.152011] ** If you see this message and you are not debugging the ** [ 0.156012] ** kernel, report this immediately to your vendor! ** [ 0.159015] ** ** [ 0.161009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ************************************************************* [ 0.167212] NET: Registered protocol family 16 [ 0.170675] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176323] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185396] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189038] cpuidle: using governor menu [ 0.191651] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193646] PCI: Using configuration type 1 for base access [ 0.195124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209130] cryptd: max_cpu_qlen set to 1000 [ 0.211454] ACPI: Added _OSI(Module Device) [ 0.212010] ACPI: Added _OSI(Processor Device) [ 0.214042] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216053] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222138] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229682] ACPI: Interpreter enabled [ 0.231055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233010] ACPI: Using IOAPIC for interrupt routing [ 0.234125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239889] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253810] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270411] acpiphp: Slot [2] registered [ 0.272136] acpiphp: Slot [5] registered [ 0.273107] acpiphp: Slot [6] registered [ 0.274256] acpiphp: Slot [7] registered [ 0.277092] acpiphp: Slot [8] registered [ 0.278102] acpiphp: Slot [9] registered [ 0.280098] acpiphp: Slot [10] registered [ 0.282159] acpiphp: Slot [3] registered [ 0.285137] acpiphp: Slot [4] registered [ 0.286074] acpiphp: Slot [11] registered [ 0.287079] acpiphp: Slot [12] registered [ 0.289078] acpiphp: Slot [13] registered [ 0.290118] acpiphp: Slot [14] registered [ 0.291127] acpiphp: Slot [15] registered [ 0.293082] acpiphp: Slot [16] registered [ 0.294095] acpiphp: Slot [17] registered [ 0.295063] acpiphp: Slot [18] registered [ 0.296184] acpiphp: Slot [19] registered [ 0.297084] acpiphp: Slot [20] registered [ 0.298076] acpiphp: Slot [21] registered [ 0.300576] acpiphp: Slot [22] registered [ 0.303110] acpiphp: Slot [23] registered [ 0.304102] acpiphp: Slot [24] registered [ 0.306165] acpiphp: Slot [25] registered [ 0.308078] acpiphp: Slot [26] registered [ 0.309077] acpiphp: Slot [27] registered [ 0.311077] acpiphp: Slot [28] registered [ 0.312117] acpiphp: Slot [29] registered [ 0.314089] acpiphp: Slot [30] registered [ 0.315097] acpiphp: Slot [31] registered [ 0.317066] PCI host bridge to bus 0000:00 [ 0.319012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340782] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345820] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362925] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372937] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376859] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379655] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417247] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.504000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.510016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.569000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.576016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606387] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612063] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.620015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.646020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.663837] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.673015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.679014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.699018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.710649] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.716011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.721013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.745026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.757463] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.760320] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.762288] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.765288] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.766171] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.770127] iommu: Default domain type: Passthrough [ 0.772780] SCSI subsystem initialized [ 0.774129] ACPI: bus type USB registered [ 0.776111] usbcore: registered new interface driver usbfs [ 0.777079] usbcore: registered new interface driver hub [ 0.779072] usbcore: registered new device driver usb [ 0.781210] pps_core: LinuxPPS API ver. 1 registered [ 0.782009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.785086] PTP clock support registered [ 0.787325] EDAC MC: Ver: 3.0.0 [ 0.789138] PCI: Using ACPI for IRQ routing [ 0.790919] NetLabel: Initializing [ 0.792009] NetLabel: domain hash size = 128 [ 0.794008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.796086] NetLabel: unlabeled traffic allowed by default [ 0.798380] vgaarb: loaded [ 0.800700] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.802012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.809298] clocksource: Switched to clocksource kvm-clock [ 0.949614] VFS: Disk quotas dquot_6.6.0 [ 0.951245] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.953553] *** VALIDATE ramfs *** [ 0.954679] *** VALIDATE hugetlbfs *** [ 0.956195] pnp: PnP ACPI init [ 0.958548] pnp: PnP ACPI: found 6 devices [ 0.975742] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.978670] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.980634] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982583] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984713] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.986833] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.989217] NET: Registered protocol family 2 [ 0.991332] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.996746] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.000288] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005772] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.009286] TCP: Hash tables configured (established 65536 bind 65536) [ 1.012106] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.015725] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019323] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.023364] NET: Registered protocol family 1 [ 1.026843] RPC: Registered named UNIX socket transport module. [ 1.028838] RPC: Registered udp transport module. [ 1.030484] RPC: Registered tcp transport module. [ 1.032325] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.034358] NET: Registered protocol family 44 [ 1.035833] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.038125] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.039867] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.041932] PCI: CLS 0 bytes, default 64 [ 1.043694] Unpacking initramfs... [ 2.859508] debug: unmapping init [mem 0xffff96bb7cc54000-0xffff96bb7ffbffff] [ 2.865980] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.868452] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.873776] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.530664] Initialise system trusted keyrings [ 3.534284] Key type blacklist registered [ 3.539062] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.585231] zbud: loaded [ 3.588262] *** VALIDATE nfs *** [ 3.589494] *** VALIDATE nfs4 *** [ 3.591223] pstore: using deflate compression [ 3.595177] Platform Keyring initialized [ 3.788440] NET: Registered protocol family 38 [ 3.791125] Key type asymmetric registered [ 3.792571] Asymmetric key parser 'x509' registered [ 3.794386] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.797695] io scheduler mq-deadline registered [ 3.799696] io scheduler kyber registered [ 3.801366] io scheduler bfq registered [ 3.803965] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.807803] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.810696] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.813624] ACPI: Power Button [PWRF] [ 3.819741] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.828293] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.850374] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.860449] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.882721] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.917377] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.958116] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.963903] Non-volatile memory driver v1.3 [ 3.966827] Linux agpgart interface v0.103 [ 4.013265] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.016846] vda: detected capacity change from 0 to 74686464 [ 4.043377] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.046806] vdb: detected capacity change from 0 to 1073741824 [ 4.072287] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.075790] vdc: detected capacity change from 0 to 2621440000 [ 4.095700] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.098663] vdd: detected capacity change from 0 to 2621440000 [ 4.112586] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.116651] vde: detected capacity change from 0 to 4294967296 [ 4.134224] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.137650] vdf: detected capacity change from 0 to 4294967296 [ 4.146497] libphy: Fixed MDIO Bus: probed [ 4.153386] usbcore: registered new interface driver usbserial_generic [ 4.156164] usbserial: USB Serial support registered for generic [ 4.158799] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.163878] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.166086] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.168792] mousedev: PS/2 mouse device common for all mice [ 4.172884] rtc_cmos 00:05: RTC can wake from S4 [ 4.179719] rtc_cmos 00:05: registered as rtc0 [ 4.181724] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.182399] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.182454] intel_pstate: CPU model not supported [ 4.187952] hid: raw HID events driver (C) Jiri Kosina [ 4.201847] usbcore: registered new interface driver usbhid [ 4.204476] usbhid: USB HID core driver [ 4.206696] drop_monitor: Initializing network drop monitor service [ 4.209281] Initializing XFRM netlink socket [ 4.212620] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.215608] NET: Registered protocol family 10 [ 4.219130] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.224267] Segment Routing with IPv6 [ 4.229563] NET: Registered protocol family 17 [ 4.231700] mpls_gso: MPLS GSO support [ 4.236596] RAS: Correctable Errors collector initialized. [ 4.238564] AVX version of gcm_enc/dec engaged. [ 4.240392] AES CTR mode by8 optimization enabled [ 4.343853] sched_clock: Marking stable (4343834458, 0)->(5467266477, -1123432019) [ 4.346790] registered taskstats version 1 [ 4.348662] Loading compiled-in X.509 certificates [ 4.350659] zswap: loaded using pool lzo/zbud [ 4.379308] Key type big_key registered [ 4.392331] Key type encrypted registered [ 4.393535] ima: No TPM chip found, activating TPM-bypass! [ 4.395394] ima: Allocated hash algorithm: sha1 [ 4.396655] ima: No architecture policies found [ 4.398200] evm: Initialising EVM extended attributes: [ 4.399503] evm: security.selinux [ 4.400412] evm: security.ima [ 4.401220] evm: security.capability [ 4.402252] evm: HMAC attrs: 0x1 [ 4.404324] rtc_cmos 00:05: setting system clock to 2026-09-02 14:15:13 UTC (1788358513) [ 4.410712] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 4.413226] debug: unmapping init [mem 0xffffffff9f782000-0xffffffff9fa58fff] [ 4.424452] Write protecting the kernel read-only data: 28672k [ 4.427545] debug: unmapping init [mem 0xffffffff9de03000-0xffffffff9dffffff] [ 4.429790] debug: unmapping init [mem 0xffffffff9e714000-0xffffffff9e7fffff] [ 4.502812] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.512567] systemd[1]: Detected virtualization kvm. [ 4.514469] systemd[1]: Detected architecture x86-64. [ 4.515708] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.548334] systemd[1]: No hostname configured. [ 4.550339] systemd[1]: Set hostname to . [ 4.552951] random: systemd: uninitialized urandom read (16 bytes read) [ 4.555833] systemd[1]: Initializing machine ID from random generator. [ 4.732802] random: ln: uninitialized urandom read (6 bytes read) [ 4.916593] random: systemd: uninitialized urandom read (16 bytes read) [ 4.919282] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.925523] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.946876] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.226462] device-mapper: uevent: version 1.0.3 [ 6.229281] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 7.438115] random: fast init done [ 7.584095] virtio_net virtio0 ens2: renamed from eth0 [ 8.149244] scsi host0: ata_piix [ 8.223169] scsi host1: ata_piix [ 8.230876] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.242607] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 15.574524] random: crng init done [ 15.575900] random: 7 urandom warning(s) missed due to ratelimiting [ 17.884346] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.290290] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.166288] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.086282] SELinux: Disabled at runtime. [ 27.228228] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.382375] systemd[1]: Detected virtualization kvm. [ 27.384452] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.686825] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.694115] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.714940] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.720992] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.726444] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.746375] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.765519] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK [ 29.891697] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 31.310665] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.778465] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.924032] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.757944] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.788921] EDAC sbridge: Ver: 1.1.2 [ 36.716712] hrtimer: interrupt took 3901300 ns [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 39.478887] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 40.780210] NFS: Registering the id_resolver key type [ 40.783742] Key type id_resolver registered [ 40.786925] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-server login: [ 104.627212] libcfs: loading out-of-tree module taints kernel. [ 104.662479] Key type ._llcrypt registered [ 104.664499] Key type .llcrypt registered [ 104.745332] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 116.269775] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 117.202824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.212248] alg: No test for adler32 (adler32-zlib) [ 118.401561] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 118.903550] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 120.575169] Key type lgssc registered [ 121.478038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.743463] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.141693] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 168.177223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.199129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.332299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.350029] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.391592] Lustre: lustre-MDT0000: new disk, initializing [ 169.427973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.437741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.629664] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.147214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.201772] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.224510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.227269] Lustre: Skipped 1 previous similar message [ 179.280659] Lustre: lustre-MDT0001: new disk, initializing [ 179.319124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.338756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.343560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.875846] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.032411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.634285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.793374] Lustre: lustre-OST0000: new disk, initializing [ 190.797422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.800815] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.842196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.417636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.427097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.470046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.196538] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.121870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.205925] Lustre: lustre-OST0001: new disk, initializing [ 204.208771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.213776] Lustre: 9496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.260863] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.575853] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.538494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.546216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.587093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.257168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.307841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.829154] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing check_logdir /tmp/testlogs/ [ 228.333095] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing yml_node [ 231.230019] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 232.711453] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 234.145702] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 235.198556] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:19:03 EDT 2026 [ 247.438899] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 248.409183] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 265.696638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.705934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.711259] Lustre: Skipped 2 previous similar messages [ 265.724354] Lustre: Skipped 3 previous similar messages [ 268.297280] Lustre: server umount lustre-MDT0000 complete [ 270.823070] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358779 with bad export cookie 6725596773339957874 [ 270.831675] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.838146] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 275.938192] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.939352] LustreError: 9450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.940509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.953978] Lustre: Skipped 2 previous similar messages [ 275.973151] LustreError: 9450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 280.031825] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.031987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.048044] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 280.051284] Lustre: Skipped 2 previous similar messages [ 285.151720] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 285.152250] LustreError: 12433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.152687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.173954] LustreError: 12433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 285.343688] Lustre: server umount lustre-MDT0001 complete [ 288.065987] Lustre: server umount lustre-OST0000 complete [ 290.496175] Lustre: server umount lustre-OST0001 complete [ 295.344558] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 301.829601] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 327.597532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 327.718914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 327.736034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 327.803711] Lustre: lustre-MDT0000: new disk, initializing [ 327.856585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.873889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 330.508805] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 338.057786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.146307] Lustre: 15039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 338.211160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 338.220845] Lustre: Skipped 1 previous similar message [ 338.286781] Lustre: lustre-MDT0001: new disk, initializing [ 338.365746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 338.394390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 338.402498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 341.139133] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 348.293318] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 352.134222] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 357.195867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.387226] Lustre: lustre-OST0000: new disk, initializing [ 357.390328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 357.394744] Lustre: 17009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 357.463804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.823143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 358.830464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 358.953586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 361.459267] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 367.275450] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 369.662609] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 369.809125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 372.276271] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 372.375434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 374.243465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 374.244979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.250197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 374.250209] Lustre: Skipped 1 previous similar message [ 374.252164] LustreError: Skipped 1 previous similar message [ 374.271700] Lustre: Skipped 1 previous similar message [ 379.502981] Lustre: server umount lustre-OST0000 complete [ 384.481906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 384.487152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.523726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.527451] Lustre: Skipped 4 previous similar messages [ 396.255763] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.389534] Lustre: server umount lustre-MDT0000 complete [ 399.845600] LustreError: 15947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.855094] LustreError: 15947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 401.281884] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358910 with bad export cookie 6725596773339959407 [ 401.283556] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.288175] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 401.582584] Lustre: server umount lustre-MDT0001 complete [ 405.459323] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:21:53 (1788358913) [ 418.040603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.344458] LustreError: 19882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.388453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.981137] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 426.120880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.973243] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 433.599476] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 437.074131] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.390140] Lustre: Failing over lustre-MDT0000 [ 438.567603] Lustre: server umount lustre-MDT0000 complete [ 441.824347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 441.825709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.829947] LustreError: 19882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.835798] Lustre: Skipped 2 previous similar messages [ 441.849208] LustreError: 19882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 456.845137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.062585] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.825859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 462.849574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 462.912279] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 464.079088] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 478.236918] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 487.393131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.415421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.415431] Lustre: Skipped 4 previous similar messages [ 487.438500] Lustre: Skipped 1 previous similar message [ 499.167652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 499.497884] Lustre: server umount lustre-MDT0000 complete [ 501.736174] LustreError: 19879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.762848] LustreError: 19879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 504.799890] LustreError: 19862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359013 with bad export cookie 6725596773339960821 [ 504.823652] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.843222] LustreError: 19862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 505.544802] Lustre: server umount lustre-MDT0001 complete [ 518.040192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.209400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.223889] Lustre: Skipped 2 previous similar messages [ 527.590546] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 534.506497] LustreError: 23222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.544947] LustreError: 23222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 543.510138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.636206] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 566.233915] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.961195] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.146869] Lustre: Failing over lustre-MDT0000 [ 577.178984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.185329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.204328] Lustre: Skipped 1 previous similar message [ 577.220847] Lustre: Skipped 6 previous similar messages [ 577.433478] Lustre: server umount lustre-MDT0000 complete [ 597.057838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.166569] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.009740] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 602.599316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 602.623323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 602.637077] Lustre: Skipped 1 previous similar message [ 602.653614] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 612.321259] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 617.959301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 617.973201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.665672] Lustre: server umount lustre-MDT0000 complete [ 624.097170] LustreError: 23969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.116392] LustreError: 23969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 627.402407] LustreError: 23204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359136 with bad export cookie 6725596773339961759 [ 627.411229] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.810412] Lustre: server umount lustre-MDT0001 complete [ 637.352552] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:25:44 (1788359144) [ 655.178274] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 666.574192] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 709.697707] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 720.309349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.504573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 720.526101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 720.586510] Lustre: lustre-MDT0000: new disk, initializing [ 720.644089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.649517] Lustre: Skipped 2 previous similar messages [ 720.659722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 725.240453] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 735.170174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.267407] Lustre: 29605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 735.462416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 739.790309] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 749.294108] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 755.157677] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 762.059704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 762.386338] Lustre: 31572:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 764.434578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 764.476440] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 764.529757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 764.544252] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 767.570800] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 776.791331] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 785.298651] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 804.614127] Lustre: server umount lustre-OST0000 complete [ 810.474984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.484805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.494663] Lustre: Skipped 2 previous similar messages [ 810.505474] Lustre: Skipped 4 previous similar messages [ 818.761506] Lustre: server umount lustre-MDT0000 complete [ 820.707851] LustreError: 29616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.726244] LustreError: 29616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 821.727371] Lustre: 3622:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359314/real 1788359314] req@ffff96bbffb22300 x1875229935894784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788359330 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 822.198966] LustreError: 29597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359331 with bad export cookie 6725596773339962165 [ 822.209822] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 822.216219] LustreError: 29597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 822.523278] Lustre: server umount lustre-MDT0001 complete [ 833.320535] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 835.669014] Key type lgssc unregistered [ 835.970966] LNet: 33548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.987892] LNetError: 33548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.007410] LNet: Removed LNI 192.168.201.125@tcp [ 836.978190] Key type .llcrypt unregistered [ 836.981419] Key type ._llcrypt unregistered [ 857.080565] Key type ._llcrypt registered [ 857.083917] Key type .llcrypt registered [ 857.184904] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 858.114168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.136555] alg: No test for adler32 (adler32-zlib) [ 859.124388] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 859.284527] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 860.951806] Key type lgssc registered [ 861.793924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.970399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.991320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.416919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.583735] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 879.588422] LustreError: 34939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.601813] LustreError: 34939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 881.161983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.476243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 884.785801] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 891.491391] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 896.091015] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 901.940786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 902.189172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.282759] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 907.496670] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 914.926368] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.397567] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:30:35 (1788359435) [ 929.654694] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 931.673531] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 938.633789] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:30:46 (1788359446) [ 947.018785] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:30:54 (1788359454) [ 948.034901] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 955.484760] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 955.487758] Lustre: Skipped 2 previous similar messages [ 956.876905] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:31:04 (1788359464) [ 959.780591] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 959.787381] Lustre: Skipped 3 previous similar messages [ 974.057499] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 974.061220] Lustre: Skipped 6 previous similar messages [ 976.375560] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:31:24 (1788359484) [ 978.735037] Lustre: *** cfs_fail_loc=131b, val=2*** [ 981.682254] Lustre: *** cfs_fail_loc=131b, val=2*** [ 989.285221] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:31:36 (1788359496) [ 990.710728] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 990.714251] Lustre: Skipped 3 previous similar messages [ 1002.480660] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:31:50 (1788359510) [ 1016.803179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.804328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1016.818982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.832107] Lustre: Skipped 1 previous similar message [ 1021.926103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.934560] Lustre: Skipped 1 previous similar message [ 1027.050514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.057305] Lustre: Skipped 1 previous similar message [ 1029.087299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1029.220941] Lustre: server umount lustre-OST0000 complete [ 1035.232621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1035.239067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.247253] Lustre: Skipped 1 previous similar message [ 1035.265400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.287867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.593275] Lustre: server umount lustre-MDT0000 complete [ 1041.804466] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359550 with bad export cookie 132442737313135826 [ 1041.808176] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.817844] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1042.222673] Lustre: server umount lustre-MDT0001 complete [ 1051.329159] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1054.010733] Key type lgssc unregistered [ 1054.303609] LNet: 42793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.316573] LNetError: 42793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.330956] LNet: Removed LNI 192.168.201.125@tcp [ 1055.264460] Key type .llcrypt unregistered [ 1055.267230] Key type ._llcrypt unregistered [ 1070.692435] Key type ._llcrypt registered [ 1070.693992] Key type .llcrypt registered [ 1070.846707] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 1084.966536] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1086.165847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.229045] alg: No test for adler32 (adler32-zlib) [ 1087.303926] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1087.508467] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1089.159269] Key type lgssc registered [ 1090.071497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1120.151718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1120.190201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.391600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1121.436206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1121.542426] Lustre: lustre-MDT0000: new disk, initializing [ 1121.629581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.644383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1125.330231] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1134.476878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.578235] Lustre: 45946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1134.629617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1134.632790] Lustre: Skipped 1 previous similar message [ 1134.697408] Lustre: lustre-MDT0001: new disk, initializing [ 1134.770986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.799034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1134.806833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1138.279507] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1145.894367] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1150.634775] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1156.273621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.570541] Lustre: lustre-OST0000: new disk, initializing [ 1156.583894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1156.596339] Lustre: 47912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1156.738438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.053027] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1165.371431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1165.378516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1165.545364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1170.925499] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1175.106958] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1175.283805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1179.073496] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1179.216921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1181.157586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.158752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.164530] LustreError: Skipped 1 previous similar message [ 1181.165556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.190676] Lustre: Skipped 1 previous similar message [ 1186.275678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.278403] Lustre: Skipped 2 previous similar messages [ 1186.817851] Lustre: server umount lustre-OST0000 complete [ 1191.394449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1191.394585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.409896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.513065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.519788] Lustre: Skipped 2 previous similar messages [ 1201.633641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.637223] Lustre: Skipped 1 previous similar message [ 1204.191375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1204.337177] Lustre: server umount lustre-MDT0000 complete [ 1207.624135] LustreError: 45937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359716 with bad export cookie 18261816006464737888 [ 1207.631195] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.633463] LustreError: 45937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1207.905423] Lustre: server umount lustre-MDT0001 complete [ 1216.016425] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:35:23 (1788359723) [ 1221.113310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.503088] LustreError: 50372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.514892] LustreError: 50372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1221.559360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.549327] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1226.731629] LustreError: 50373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.355195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.672763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.117525] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1240.635588] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1244.413336] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1248.386069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.595274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.216316] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1259.228851] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1272.627390] LustreError: 52925:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1273.289419] Lustre: Modifying parameter general.timeout=1 in log params [ 1274.543635] Lustre: Modifying parameter general.timeout=3 in log params [ 1274.546574] Lustre: Skipped 1 previous similar message [ 1277.030822] Lustre: Modifying parameter general.timeout=7 in log params [ 1277.033209] Lustre: Skipped 3 previous similar messages [ 1281.564492] Lustre: Modifying parameter general.timeout=14 in log params [ 1281.567678] Lustre: Skipped 6 previous similar messages [ 1290.060645] Lustre: Modifying parameter general.timeout=25 in log params [ 1290.063593] Lustre: Skipped 10 previous similar messages [ 1306.480472] Lustre: Modifying parameter general.timeout=48 in log params [ 1306.483457] Lustre: Skipped 22 previous similar messages [ 1338.573210] Lustre: Modifying parameter general.timeout=95 in log params [ 1338.578666] Lustre: Skipped 46 previous similar messages [ 1371.168559] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:37:59 (1788359879) [ 1373.319941] Lustre: 59849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1374.527640] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1377.237650] Lustre: 60159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1377.243525] Lustre: 60159:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1398.944947] Lustre: 61711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1398.953573] Lustre: 61711:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1398.980639] LustreError: 61711:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1402.354707] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:38:30 (1788359910) [ 1405.141411] Lustre: Setting parameter general.jobid_var=TESTNAME in log params [ 1405.143970] Lustre: Skipped 56 previous similar messages [ 1410.581838] Lustre: server umount lustre-MDT0000 complete [ 1411.042789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1411.052812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.069255] Lustre: Skipped 1 previous similar message [ 1411.075867] LustreError: 51099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.010455] LustreError: 51106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359922 with bad export cookie 18261816006464738770 [ 1413.017587] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.028057] LustreError: 51106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1413.089536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.092827] LustreError: 50367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.102196] Lustre: Skipped 1 previous similar message [ 1413.103081] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1418.211142] LustreError: 56662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.352924] Lustre: server umount lustre-MDT0001 complete [ 1422.147632] Lustre: server umount lustre-OST0000 complete [ 1442.113600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.318931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1442.544349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.804979] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1452.316545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.373244] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1452.412721] Lustre: 64235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1455.632241] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1461.228367] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1465.321772] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1470.039258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1470.141870] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1474.321922] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1478.819745] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1479.940150] Lustre: 66800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1486.822590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.823143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.825653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.825663] Lustre: Skipped 1 previous similar message [ 1486.830792] Lustre: Skipped 1 previous similar message [ 1486.834974] LustreError: Skipped 1 previous similar message [ 1491.061148] Lustre: server umount lustre-OST0000 complete [ 1497.062852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.080794] Lustre: Skipped 1 previous similar message [ 1508.319179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1508.573557] Lustre: server umount lustre-MDT0000 complete [ 1511.516933] LustreError: 64227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360020 with bad export cookie 18261816006464741549 [ 1511.517882] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1511.527764] LustreError: 64227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1511.781541] Lustre: server umount lustre-MDT0001 complete [ 1519.808909] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1521.623938] Key type lgssc unregistered [ 1521.858834] LNet: 68005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.870104] LNetError: 68005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.878800] LNet: Removed LNI 192.168.201.125@tcp [ 1522.574680] Key type .llcrypt unregistered [ 1522.578195] Key type ._llcrypt unregistered [ 1539.974023] Key type ._llcrypt registered [ 1539.982171] Key type .llcrypt registered [ 1540.068324] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1540.781111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.845361] alg: No test for adler32 (adler32-zlib) [ 1541.852460] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1542.074585] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1543.735236] Key type lgssc registered [ 1544.561913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1550.539622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1550.556286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.947871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.534239] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1561.059119] LustreError: 69397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.084962] LustreError: 69397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1562.079432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.432849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1564.904250] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1569.517586] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1573.881265] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1578.377951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.744637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.934403] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1588.320530] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1603.963364] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1612.773534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.779164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.783407] Lustre: Skipped 1 previous similar message [ 1615.967261] Lustre: server umount lustre-MDT0000 complete [ 1617.894130] LustreError: 69393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.906302] LustreError: 69393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1618.288763] LustreError: 69377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360127 with bad export cookie 9830701259580386552 [ 1618.291963] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.298598] LustreError: 69377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1618.496397] Lustre: server umount lustre-MDT0001 complete [ 1631.465813] Lustre: server umount lustre-OST0000 complete [ 1649.734809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.926409] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1649.956268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1650.112677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.745563] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1659.070314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.150349] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1659.178783] Lustre: 74289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1659.518726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.473528] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1667.837156] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1672.131134] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1676.818664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1676.956194] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1677.207667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.602353] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1687.088170] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.533729] Lustre: 76860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.046252] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1697.253096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1697.261053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.275122] Lustre: Skipped 1 previous similar message [ 1697.284554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.290935] Lustre: Skipped 2 previous similar messages [ 1697.766901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.585626] Lustre: server umount lustre-OST0000 complete [ 1702.884014] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.898390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.902081] Lustre: Skipped 2 previous similar messages [ 1702.916816] Lustre: Skipped 2 previous similar messages [ 1708.000160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1708.006606] Lustre: Skipped 1 previous similar message [ 1708.558207] Lustre: server umount lustre-MDT0000 complete [ 1711.345225] LustreError: 74281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360220 with bad export cookie 9830701259580387392 [ 1711.354296] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.356137] LustreError: 74281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1711.573596] Lustre: server umount lustre-MDT0001 complete [ 1719.070975] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1721.145562] Key type lgssc unregistered [ 1721.368606] LNet: 78070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.371876] LNetError: 78070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.386091] LNet: Removed LNI 192.168.201.125@tcp [ 1722.059323] Key type .llcrypt unregistered [ 1722.061595] Key type ._llcrypt unregistered [ 1740.569463] Key type ._llcrypt registered [ 1740.571325] Key type .llcrypt registered [ 1740.641808] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1741.337816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.389534] alg: No test for adler32 (adler32-zlib) [ 1742.351482] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1742.567055] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1744.168402] Key type lgssc registered [ 1744.921991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.824616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.853611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.263542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.011631] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1760.884866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.283074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1764.100630] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1769.403307] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1773.349437] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1778.247876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.605803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.806978] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1787.534187] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.849791] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1803.501438] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1804.715617] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1804.719697] Lustre: Skipped 1 previous similar message [ 1807.197249] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1807.201610] Lustre: Skipped 3 previous similar messages [ 1811.693389] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1811.697152] Lustre: Skipped 6 previous similar messages [ 1819.971802] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1819.976203] Lustre: Skipped 12 previous similar messages [ 1836.013976] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1836.017634] Lustre: Skipped 24 previous similar messages [ 1867.949974] Lustre: 87566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1868.633171] Lustre: Disabling parameter general.jobid_var= in log params [ 1868.639165] Lustre: Skipped 47 previous similar messages [ 1869.650203] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:46:17 (1788360377) [ 1870.520054] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1871.544815] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:46:19 (1788360379) [ 1873.890592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.892089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.913474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1879.015158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1879.023599] Lustre: Skipped 2 previous similar messages [ 1879.558550] Lustre: server umount lustre-OST0000 complete [ 1884.128632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.132152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.139114] LustreError: Skipped 1 previous similar message [ 1884.140160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.145416] Lustre: Skipped 2 previous similar messages [ 1884.153336] Lustre: Skipped 1 previous similar message [ 1887.960151] Lustre: server umount lustre-MDT0000 complete [ 1889.248377] LustreError: 80275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.258341] LustreError: 80275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1890.084138] LustreError: 79443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360399 with bad export cookie 13056165595639061910 [ 1890.086273] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.088939] LustreError: 79443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1890.261702] Lustre: server umount lustre-MDT0001 complete [ 1896.141221] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1897.665725] Key type lgssc unregistered [ 1897.842703] LNet: 88811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.849865] LNetError: 88811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.919282] LNet: Removed LNI 192.168.201.125@tcp [ 1899.464062] Key type .llcrypt unregistered [ 1899.466229] Key type ._llcrypt unregistered [ 1902.317837] Key type ._llcrypt registered [ 1902.320342] Key type .llcrypt registered [ 1902.384477] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1905.086105] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules [ 1905.833666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.848382] alg: No test for adler32 (adler32-zlib) [ 1906.782599] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1906.910983] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1908.536230] Key type lgssc registered [ 1909.139262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.900095] LustreError: 90003:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1921.434428] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1926.988866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1932.045751] LustreError: 90003:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1932.104052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.556573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.423258] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1942.417670] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:47:30 (1788360450) [ 1943.008556] LustreError: 91039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.025300] LustreError: 91039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1945.804185] LustreError: 91070:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1945.820645] LustreError: 91070:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1945.825496] LustreError: 91070:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1946.056386] Lustre: server umount lustre-MDT0000 complete [ 1954.086263] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1955.833583] Key type lgssc unregistered [ 1956.018315] LNet: 92561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.027495] LNetError: 92561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.042578] LNet: Removed LNI 192.168.201.125@tcp [ 1956.551937] Key type .llcrypt unregistered [ 1956.557357] Key type ._llcrypt unregistered [ 1968.326836] Key type ._llcrypt registered [ 1968.329101] Key type .llcrypt registered [ 1968.399253] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1969.300314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.325702] alg: No test for adler32 (adler32-zlib) [ 1970.320429] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1970.487570] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1972.111174] Key type lgssc registered [ 1972.956929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.171036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.188249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.558146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.375269] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1988.162621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.555174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.423324] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1996.766885] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.808287] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2005.529980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2005.965989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.216630] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2015.445388] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2060.176814] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:49:28 (1788360568) [ 2061.337533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2062.694789] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:49:30 (1788360570) [ 2065.379037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2065.380769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.388158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.399639] Lustre: Skipped 2 previous similar messages [ 2070.496935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.501717] Lustre: Skipped 4 previous similar messages [ 2071.411879] Lustre: server umount lustre-MDT0000 complete [ 2074.116541] LustreError: 93739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360583 with bad export cookie 16810461279837365235 [ 2074.116994] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.121178] LustreError: 93739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2074.333666] Lustre: server umount lustre-MDT0001 complete [ 2087.963501] Lustre: server umount lustre-OST0000 complete [ 2112.078632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.652879] LustreError: 98045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.674163] LustreError: 98045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2112.788600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.119330] LustreError: 98046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.050746] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2122.213465] LustreError: 98045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.331454] LustreError: 98046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.459633] LustreError: 98045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.717646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.715453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.399438] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2151.992637] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.931279] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2171.776841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2172.077378] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2172.088439] LustreError: 98033:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2172.110723] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2172.117616] LustreError: 100131:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2172.129426] LustreError: 100131:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2172.132697] LustreError: 100131:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2172.234082] Lustre: server umount lustre-OST0000 complete [ 2172.237307] LustreError: 100131:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2180.843988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.051630] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2181.065719] LustreError: 98036:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2181.082832] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2181.104658] LustreError: 100592:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2181.123422] LustreError: 100592:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2181.136516] LustreError: 100592:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2181.206880] Lustre: server umount lustre-OST0000 complete [ 2181.210215] LustreError: 100592:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2190.535775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.703554] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2190.708814] Lustre: Found index 0 for lustre-OST0000, updating log [ 2190.715991] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2190.782655] Lustre: lustre-OST0000: new disk, initializing [ 2190.785905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2190.791652] Lustre: 101127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2191.090091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.356736] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2201.036814] Lustre: Failing over lustre-OST0000 [ 2201.196707] Lustre: server umount lustre-OST0000 complete [ 2204.641832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2204.646923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.672737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.228793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.230349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.739587] Lustre: server umount lustre-MDT0000 complete [ 2213.347425] LustreError: 98041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.359427] LustreError: 98041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2213.974289] LustreError: 98026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360723 with bad export cookie 16810461279837367986 [ 2213.985763] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.990375] LustreError: 98026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2214.332111] Lustre: server umount lustre-MDT0001 complete [ 2222.724422] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:52:10 (1788360730) [ 2239.260243] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2249.492804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.183652] LustreError: 103471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.311966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.690393] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2263.148338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.666323] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2270.332462] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2277.132129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2284.178439] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2292.958662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.070415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2293.123723] Lustre: lustre-OST0001: new disk, initializing [ 2293.127343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2293.132034] Lustre: 105788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2293.407454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2293.419869] Lustre: Skipped 2 previous similar messages [ 2295.199532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2295.225301] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2295.341771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2300.267962] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2307.841796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2313.396865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2318.355313] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [ 2318.358517] Lustre: Skipped 1 previous similar message [ 2320.356036] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.375150] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.671680] Lustre: server umount lustre-MDT0001 complete [ 2327.016792] LustreError: 104203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.054219] LustreError: 104203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2332.561260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.696516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2332.702108] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2332.749220] Lustre: 103459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2334.355202] LustreError: 107359:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2334.367176] LustreError: 107359:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2334.389383] LustreError: 103464:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2334.601965] LustreError: 107374:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -17 [ 2334.613908] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2334.613908] [ 2337.770325] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2338.803519] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2347.305129] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:54:14 (1788360854) [ 2366.475535] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2383.167348] Lustre: 109406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2404.389856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2407.683452] Lustre: 110540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2491.874522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2491.882768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.891775] Lustre: Skipped 2 previous similar messages [ 2491.895717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2491.899636] Lustre: Skipped 5 previous similar messages [ 2497.037728] Lustre: server umount lustre-MDT0000 complete [ 2497.508214] LustreError: 106317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.521676] LustreError: 106317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2500.137521] LustreError: 103453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361009 with bad export cookie 16810461279837368910 [ 2500.140912] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.142825] LustreError: 103453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2500.494378] Lustre: server umount lustre-MDT0001 complete [ 2514.336934] Lustre: server umount lustre-OST0000 complete [ 2527.265295] Lustre: server umount lustre-OST0001 complete [ 2532.430590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2540.905158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2547.532663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2555.116807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2570.730687] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2578.569140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.589170] Lustre: lustre-MDT0000: reset Object Index mappings [ 2578.952860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.956061] Lustre: Skipped 1 previous similar message [ 2582.482639] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2589.643607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.663853] Lustre: lustre-MDT0001: reset Object Index mappings [ 2594.390829] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2597.152505] Lustre: 115639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2603.138278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.943571] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2615.090376] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2615.579844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.795557] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2621.434849] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2623.481573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2623.491070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2632.402845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2635.402836] Lustre: 117515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2680.228566] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:59:47 (1788361187) [ 2707.936470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2707.949296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.962539] Lustre: Skipped 3 previous similar messages [ 2707.975551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.978030] Lustre: Skipped 4 previous similar messages [ 2714.614107] Lustre: server umount lustre-MDT0000 complete [ 2715.617272] LustreError: 114488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.630631] LustreError: 114488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2717.832332] LustreError: 114473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361226 with bad export cookie 16810461279837415782 [ 2717.838905] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.851602] LustreError: 114473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.282654] Lustre: server umount lustre-MDT0001 complete [ 2742.242342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2742.340421] Lustre: server umount lustre-OST0000 complete [ 2751.697530] Lustre: server umount lustre-OST0001 complete [ 2756.726549] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 2765.146986] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2793.660780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.831483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2793.848471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2793.927303] Lustre: lustre-MDT0000: new disk, initializing [ 2793.977217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.983640] Lustre: Skipped 3 previous similar messages [ 2793.993312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2797.157265] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2803.227261] Lustre: server umount lustre-MDT0000 complete [ 2811.257508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.474944] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2811.486511] Lustre: 121790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2811.497753] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2814.977327] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2818.689360] Lustre: server umount lustre-MDT0000 complete [ 2826.030561] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:02:13 (1788361333) [ 2827.147843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2828.482642] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:02:16 (1788361336) [ 2844.802073] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2854.900393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.992803] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2866.156207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.223495] Lustre: 123846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2866.262050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2866.265438] Lustre: Skipped 1 previous similar message [ 2866.353918] Lustre: lustre-MDT0001: new disk, initializing [ 2866.449302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2866.456179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2870.598747] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2874.917947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2880.168734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.395363] Lustre: lustre-OST0000: new disk, initializing [ 2880.397555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2880.401781] Lustre: 125425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2881.931272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2881.938153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2886.304521] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2887.168862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2895.109780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.236297] Lustre: lustre-OST0001: new disk, initializing [ 2895.240219] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2895.245855] Lustre: 126298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2896.647381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2896.656925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2896.732871] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2900.174185] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2908.316132] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2911.224658] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3188.408623] Lustre: 132690:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3188.413456] Lustre: 132690:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3188.416426] Lustre: 132690:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3188.419470] Lustre: 132690:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3188.424332] Lustre: 132690:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3188.427301] Lustre: 132690:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3400.271972] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:11:47 (1788361907) [ 3413.984500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.996678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.006433] Lustre: Skipped 8 previous similar messages [ 3414.014763] Lustre: Skipped 11 previous similar messages [ 3419.109923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.118437] Lustre: Skipped 3 previous similar messages [ 3419.612826] Lustre: server umount lustre-MDT0000 complete [ 3422.274537] LustreError: 132703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361931 with bad export cookie 16810461279837481631 [ 3422.275513] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3422.285090] LustreError: 132703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3424.226599] LustreError: 127101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.227961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.244930] LustreError: 127101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3424.260645] Lustre: Skipped 1 previous similar message [ 3427.232703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3427.239325] Lustre: Skipped 3 previous similar messages [ 3428.671672] Lustre: server umount lustre-MDT0001 complete [ 3437.302801] Lustre: server umount lustre-OST0000 complete [ 3445.930954] Lustre: server umount lustre-OST0001 complete [ 3450.724481] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3457.163386] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3480.678550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.840780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3480.861808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3480.920788] Lustre: lustre-MDT0000: new disk, initializing [ 3480.964597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.968098] Lustre: Skipped 5 previous similar messages [ 3480.976254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3483.704237] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3491.575459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.654803] Lustre: 135725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3491.821304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3495.016822] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3502.757060] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.201204] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3511.952942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.147115] Lustre: 137693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3513.345469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3513.353425] Lustre: Skipped 1 previous similar message [ 3513.357764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3513.436173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.148696] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3523.244488] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3530.173536] Lustre: lustre-MDD0000: changelog on [ 3532.257310] Lustre: lustre-MDD0001: changelog on [ 4919.668807] Lustre: lustre-MDD0001: changelog off [ 4920.872274] Lustre: 135732:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4921.524564] Lustre: lustre-MDD0000: changelog off [ 4925.920371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4925.926190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.944489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.433549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.554181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4931.558572] Lustre: Skipped 3 previous similar messages [ 4936.676818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.681640] Lustre: Skipped 1 previous similar message [ 4939.231117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4939.372353] Lustre: server umount lustre-OST0000 complete [ 4941.792842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.803287] Lustre: Skipped 1 previous similar message [ 4946.919735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.923734] Lustre: Skipped 3 previous similar messages [ 4947.957583] Lustre: server umount lustre-MDT0000 complete [ 4950.336504] LustreError: 137692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363459 with bad export cookie 16810461279838292616 [ 4950.350401] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.648331] Lustre: server umount lustre-MDT0001 complete [ 4956.757629] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4958.327621] Key type lgssc unregistered [ 4958.498701] LNet: 146755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4958.503940] LNetError: 146755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4958.518739] LNet: Removed LNI 192.168.201.125@tcp [ 4958.955286] Key type .llcrypt unregistered [ 4958.957054] Key type ._llcrypt unregistered [ 4971.233917] Key type ._llcrypt registered [ 4971.235173] Key type .llcrypt registered [ 4971.296249] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4979.969304] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4980.513557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4980.524609] alg: No test for adler32 (adler32-zlib) [ 4981.481211] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4981.653032] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4983.271169] Key type lgssc registered [ 4983.920786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5004.638628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5004.650572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.753149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.782601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5005.845167] Lustre: lustre-MDT0000: new disk, initializing [ 5005.892886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.902580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5008.317732] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5014.899754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.968239] Lustre: 149931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5014.999478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5015.002210] Lustre: Skipped 1 previous similar message [ 5015.056396] Lustre: lustre-MDT0001: new disk, initializing [ 5015.117106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.145996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5015.155737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5018.027684] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5024.061428] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5027.166383] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.162605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.322395] Lustre: lustre-OST0000: new disk, initializing [ 5031.325634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5031.332610] Lustre: 151896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5031.394344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.606925] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5038.615315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5038.623709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5038.690158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5040.237430] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5043.318767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_module obdecho/obdecho [ 5044.277993] LustreError: 152900:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5044.282695] LustreError: 152900:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5045.201596] LustreError: 152996:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5053.924992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.930168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.932338] Lustre: Skipped 1 previous similar message [ 5053.945495] Lustre: Skipped 2 previous similar messages [ 5057.248781] Lustre: server umount lustre-MDT0000 complete [ 5059.143680] LustreError: 149924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363568 with bad export cookie 3486303665388554480 [ 5059.146204] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5059.150992] LustreError: 149924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5059.303590] Lustre: server umount lustre-MDT0001 complete [ 5071.531839] Lustre: server umount lustre-OST0000 complete [ 5083.298110] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5088.606337] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5107.347865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.471642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5107.486749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5107.535455] Lustre: lustre-MDT0000: new disk, initializing [ 5107.584958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.594910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5109.813044] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5115.700559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.748967] Lustre: 156705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5115.781225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5115.787752] Lustre: Skipped 1 previous similar message [ 5115.830804] Lustre: lustre-MDT0001: new disk, initializing [ 5115.881409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5115.904170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5115.907761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5117.906384] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5123.063443] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5125.593613] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5128.714467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.831715] Lustre: lustre-OST0000: new disk, initializing [ 5128.834570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5128.838381] Lustre: 158675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5128.881541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.191846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5130.202038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5130.257423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5131.753626] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5136.512833] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5138.421939] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5138.512418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5140.329592] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5140.421989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5145.571316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5145.574965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.577564] LustreError: Skipped 1 previous similar message [ 5145.578246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.598995] Lustre: Skipped 2 previous similar messages [ 5147.354038] Lustre: server umount lustre-OST0000 complete [ 5150.688967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.689459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.694245] Lustre: Skipped 1 previous similar message [ 5150.696810] Lustre: Skipped 1 previous similar message [ 5155.164538] Lustre: server umount lustre-MDT0000 complete [ 5155.808766] LustreError: 157932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.819945] LustreError: 157932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5157.007344] LustreError: 156697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363666 with bad export cookie 3486303665388555439 [ 5157.010706] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5157.013152] LustreError: 156697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5157.192284] Lustre: server umount lustre-MDT0001 complete [ 5158.959591] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:41:07 (1788363667) [ 5159.622657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5160.417237] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:41:08 (1788363668) [ 5164.095730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.381525] LustreError: 161083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.414294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.198049] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5169.635433] LustreError: 161084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.000864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.064559] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5175.470190] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5178.221839] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5181.185486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.832379] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5186.856397] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5200.865387] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.867143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5200.870030] Lustre: Skipped 2 previous similar messages [ 5200.872127] Lustre: Skipped 2 previous similar messages [ 5200.876280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5203.049416] Lustre: server umount lustre-MDT0000 complete [ 5204.847840] LustreError: 162925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363713 with bad export cookie 3486303665388556314 [ 5204.851652] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.855647] LustreError: 162925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5205.984256] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.984499] LustreError: 161079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.989981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5206.000144] Lustre: Skipped 2 previous similar messages [ 5210.015719] LustreError: 161080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.022147] LustreError: 161080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5214.112711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5214.115736] Lustre: Skipped 3 previous similar messages [ 5219.295243] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5219.296184] LustreError: 163571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.305632] LustreError: 163571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5219.391780] Lustre: server umount lustre-MDT0001 complete [ 5222.125069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5227.679829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.783457] LustreError: 164942:0:(llog.c:1655:llog_backup()) MGC192.168.201.125@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5227.787404] Lustre: 164942:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.125@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5248.511219] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.325916] LustreError: 164949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.333181] LustreError: 164949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 5288.927277] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5288.929339] Lustre: 164942:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5329.911641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5329.916779] Lustre: Skipped 2 previous similar messages [ 5331.866749] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5335.013031] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5336.141407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.862185] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.868711] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 5349.344877] LustreError: 148157:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005d7f82ec x1875235034999936/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5351.437229] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5354.585570] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5354.981576] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5355.275685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5357.525141] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5360.307435] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5370.337217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5370.337893] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.341656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.347260] Lustre: Skipped 1 previous similar message [ 5370.349922] Lustre: Skipped 2 previous similar messages [ 5370.960869] Lustre: server umount lustre-MDT0000 complete [ 5372.510260] LustreError: 162925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363881 with bad export cookie 3486303665388562768 [ 5372.513543] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5372.515150] LustreError: 162925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5372.652836] Lustre: server umount lustre-MDT0001 complete [ 5384.420379] Lustre: server umount lustre-OST0000 complete [ 5392.541095] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5396.476844] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5411.186302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.285079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.297197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.335500] Lustre: lustre-MDT0000: new disk, initializing [ 5411.379642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.382591] Lustre: Skipped 1 previous similar message [ 5411.388785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5413.091863] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5418.172161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.223962] Lustre: 170013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5418.347553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5420.288038] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5425.311782] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5427.914780] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5431.041570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5431.166882] Lustre: 171982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5431.292641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5431.347333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5433.987779] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5438.600189] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5440.478329] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5440.588579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5442.339702] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5442.430932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5446.626473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.627019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.628530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.628537] Lustre: Skipped 2 previous similar messages [ 5446.636970] LustreError: Skipped 1 previous similar message [ 5446.645458] Lustre: Skipped 2 previous similar messages [ 5449.435328] Lustre: server umount lustre-OST0000 complete [ 5457.121680] Lustre: server umount lustre-MDT0000 complete [ 5458.678299] LustreError: 170807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363967 with bad export cookie 3486303665388579596 [ 5458.680944] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5458.683166] LustreError: 170807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5458.815373] Lustre: server umount lustre-MDT0001 complete [ 5462.706287] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:46:11 (1788363971) [ 5466.149436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.379511] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5472.421257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.630290] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5478.222697] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.804523] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5483.908337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5486.774121] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5489.838519] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5504.356341] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:46:52 (1788364012) [ 5506.528596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5506.534925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.550404] Lustre: Skipped 2 previous similar messages [ 5512.088355] Lustre: server umount lustre-OST0000 complete [ 5517.794198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.797830] Lustre: Skipped 9 previous similar messages [ 5528.031326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5528.111495] Lustre: server umount lustre-MDT0000 complete [ 5529.953053] LustreError: 174372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364039 with bad export cookie 3486303665388580478 [ 5529.961156] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5529.972089] LustreError: 174372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5530.148635] Lustre: server umount lustre-MDT0001 complete [ 5535.132449] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5536.332197] Key type lgssc unregistered [ 5536.467413] LNet: 178136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5536.470332] LNetError: 178136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5536.484538] LNet: Removed LNI 192.168.201.125@tcp [ 5536.839936] Key type .llcrypt unregistered [ 5536.841496] Key type ._llcrypt unregistered [ 5546.060579] Key type ._llcrypt registered [ 5546.062285] Key type .llcrypt registered [ 5546.112606] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5546.534664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5546.588109] alg: No test for adler32 (adler32-zlib) [ 5547.522921] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5547.655079] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5549.247158] Key type lgssc registered [ 5549.723343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5553.058301] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5553.073454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5568.543663] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5573.663382] LustreError: 179356:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 5578.783321] LustreError: 179356:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5578.787637] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5578.789637] LustreError: MGC192.168.201.125@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5578.794507] LustreError: 179356:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5578.797687] LustreError: 179356:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5578.800415] LustreError: 179356:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5578.820316] Lustre: server umount lustre-OST0000 complete [ 5578.822130] LustreError: 179356:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5583.021776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.214780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.930528] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5590.442835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.595412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.498314] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5595.790346] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5598.492237] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.637648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.721034] LustreError: 181669:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5601.730151] Lustre: 181669:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.125@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5601.789647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.274902] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5607.293030] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5608.178997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.180621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.183748] Lustre: Skipped 2 previous similar messages [ 5608.187943] Lustre: Skipped 2 previous similar messages [ 5613.537511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.539787] Lustre: Skipped 2 previous similar messages [ 5614.319339] Lustre: server umount lustre-MDT0000 complete [ 5615.585801] LustreError: 179856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.593955] LustreError: 179856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5616.026601] LustreError: 179836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364125 with bad export cookie 4826325458755025350 [ 5616.030560] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.031044] LustreError: 179836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.631288] LustreError: 180583:0:(client.c:1384:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000091c958c2 x1875235628404992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5617.695366] Lustre: server umount lustre-MDT0001 complete [ 5629.557042] Lustre: server umount lustre-OST0000 complete [ 5633.635229] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5637.822401] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5653.483559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.577115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5653.587821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5653.625441] Lustre: lustre-MDT0000: new disk, initializing [ 5653.655726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.664594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5655.323945] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5660.185946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.240637] Lustre: 185185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5660.282315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5660.285432] Lustre: Skipped 1 previous similar message [ 5660.324990] Lustre: lustre-MDT0001: new disk, initializing [ 5660.379779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5660.408475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5660.419155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5662.048483] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5666.833404] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5669.557706] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5672.721330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5672.868941] Lustre: lustre-OST0000: new disk, initializing [ 5672.872500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5672.876184] Lustre: 187152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5672.922746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.240524] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5679.097660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5679.101959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5679.139118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5679.735052] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5681.743405] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5681.825901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5683.482594] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5683.569897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5686.591502] Lustre: server umount lustre-OST0000 complete [ 5689.312605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.313398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.317466] Lustre: Skipped 1 previous similar message [ 5694.432600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5694.434590] Lustre: Skipped 2 previous similar messages [ 5698.575901] Lustre: server umount lustre-MDT0000 complete [ 5699.552947] LustreError: 185194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.564348] LustreError: 185194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5700.305332] LustreError: 187151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364209 with bad export cookie 4826325458755026036 [ 5700.305848] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.312078] LustreError: 187151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5710.873109] Lustre: server umount lustre-MDT0001 complete [ 5714.557943] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:50:23 (1788364223) [ 5721.312356] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5722.455785] Key type lgssc unregistered [ 5722.607347] LNet: 190093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5722.611374] LNetError: 190093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5722.620459] LNet: Removed LNI 192.168.201.125@tcp [ 5722.954571] Key type .llcrypt unregistered [ 5722.956054] Key type ._llcrypt unregistered [ 5730.876295] Key type ._llcrypt registered [ 5730.877678] Key type .llcrypt registered [ 5730.921702] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5731.290185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5731.335901] alg: No test for adler32 (adler32-zlib) [ 5732.220447] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5732.326190] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5733.911154] Key type lgssc registered [ 5734.354524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5737.198051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5737.205573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.371151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.939903] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5743.470717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.618418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5745.299662] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5748.162552] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5750.376568] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5753.453238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5753.540472] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5753.542146] LustreError: 193152:0:(llog.c:1696:llog_backup()) MGC192.168.201.125@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5753.546747] Lustre: 193152:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.125@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5753.604063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.174198] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5759.100264] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5759.970938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.973966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.975413] Lustre: Skipped 2 previous similar messages [ 5759.977524] Lustre: Skipped 1 previous similar message [ 5765.090703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.093384] Lustre: Skipped 2 previous similar messages [ 5770.211278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.213706] Lustre: Skipped 2 previous similar messages [ 5774.303134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5774.360965] Lustre: server umount lustre-MDT0000 complete [ 5775.330944] LustreError: 191288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.336225] LustreError: 191288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5775.969316] LustreError: 192022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364285 with bad export cookie 14672378455221944352 [ 5775.972020] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.973296] LustreError: 192022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5776.105876] Lustre: server umount lustre-MDT0001 complete [ 5787.938807] Lustre: server umount lustre-OST0000 complete [ 5791.676789] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5795.914981] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5810.899709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.976147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5810.987898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5811.021136] Lustre: lustre-MDT0000: new disk, initializing [ 5811.042565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.049214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5812.495216] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5817.144650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.194988] Lustre: 196669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5817.217655] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5817.219877] Lustre: Skipped 1 previous similar message [ 5817.222053] LustreError: 197454:0:(llog.c:1696:llog_backup()) MGC192.168.201.125@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5817.225432] LustreError: 197454:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5817.228862] Lustre: 197454:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.125@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5817.239834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5817.242933] Lustre: Skipped 1 previous similar message [ 5817.282094] Lustre: lustre-MDT0001: new disk, initializing [ 5817.321803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5817.334066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5817.338066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5818.892814] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5823.311656] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5825.538376] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.218644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.338538] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5828.340384] Lustre: Skipped 1 previous similar message [ 5828.342370] LustreError: 198614:0:(llog.c:1696:llog_backup()) MGC192.168.201.125@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5828.347050] LustreError: 198614:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5828.349781] Lustre: 198614:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.125@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5828.356360] Lustre: 198614:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5828.371127] Lustre: lustre-OST0000: new disk, initializing [ 5828.373673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5828.376613] Lustre: 198638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5828.411119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.746460] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5834.918693] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5836.587912] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5837.315364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5837.319920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5837.356197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5837.710316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5839.366500] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5839.459866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5842.401040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5842.403131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.407441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5842.411734] Lustre: Skipped 1 previous similar message [ 5842.416447] Lustre: Skipped 1 previous similar message [ 5846.354975] Lustre: server umount lustre-OST0000 complete [ 5848.031774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5848.035117] LustreError: Skipped 1 previous similar message [ 5848.036519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.041123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5852.641712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.760399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5857.765709] Lustre: Skipped 3 previous similar messages [ 5862.367137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5862.449989] Lustre: server umount lustre-MDT0000 complete [ 5862.882927] LustreError: 197482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.891883] LustreError: 197482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5864.063111] LustreError: 196663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364373 with bad export cookie 14672378455221945052 [ 5864.066709] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5864.068211] LustreError: 196663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5864.228887] Lustre: server umount lustre-MDT0001 complete [ 5867.875466] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:52:56 (1788364376) [ 5874.467420] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5878.895755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.072245] LustreError: 201770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.105215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.878610] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5884.384249] LustreError: 201771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.911041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.651087] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5887.901581] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5890.410943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5892.874144] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5896.873512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.951517] Lustre: lustre-OST0001: new disk, initializing [ 5896.953463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5896.956728] Lustre: 204079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5900.082123] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5903.440177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5903.445124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5903.540168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5911.602567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5915.167844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5916.661907] Lustre: DEBUG MARKER: ADD OST3 [ 5920.227729] loop: module loaded [ 5920.278657] loop0: detected capacity change from 0 to 204800000 [ 5920.366159] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5920.374514] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5920.384499] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5924.517558] loop0: detected capacity change from 0 to 204800000 [ 5926.407268] Lustre: DEBUG MARKER: STOP OST3 [ 5928.431459] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5928.526225] Lustre: lustre-OST0003: new disk, initializing [ 5928.528491] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5928.532703] Lustre: 206009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5928.584638] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5928.588185] Lustre: Skipped 3 previous similar messages [ 5930.399141] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5930.418815] Lustre: Failing over lustre-OST0003 [ 5930.461469] LustreError: 203256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.530643] Lustre: server umount lustre-OST0003 complete [ 5932.448170] LustreError: 206231:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5933.156515] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5935.631181] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.780592] Lustre: DEBUG MARKER: START OST3 again [ 5940.252342] LustreError: 203255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.260969] LustreError: 203255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5942.734659] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5942.866966] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5944.142691] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5944.818351] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5944.819263] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5944.833811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5944.841826] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5945.888330] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5947.923073] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5961.361167] Lustre: Failing over lustre-OST0003 [ 5961.433345] Lustre: server umount lustre-OST0003 complete [ 5963.232606] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.237770] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.239777] LustreError: 203256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.243480] Lustre: Skipped 1 previous similar message [ 5963.252227] LustreError: 203256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5977.569056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5977.575174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.367400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5990.518945] Lustre: server umount lustre-MDT0000 complete [ 5992.676126] LustreError: 203278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364501 with bad export cookie 14672378455221945920 [ 5992.678863] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.686178] LustreError: 203278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5993.958908] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.977804] Lustre: Skipped 5 previous similar messages [ 5997.984937] LustreError: 201767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.999073] LustreError: 201767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5998.956758] Lustre: server umount lustre-MDT0001 complete [ 6001.256789] Lustre: server umount lustre-OST0000 complete [ 6003.362419] Lustre: server umount lustre-OST0001 complete [ 6007.085357] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6011.504510] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6028.677798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.771173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6028.784384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6028.820234] Lustre: lustre-MDT0000: new disk, initializing [ 6028.849069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.851558] Lustre: Skipped 1 previous similar message [ 6028.857646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6030.316955] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6035.126295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.181815] Lustre: 211378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6035.330363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6037.128958] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6041.745161] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6044.222212] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6047.225093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6047.406265] Lustre: lustre-OST0000: new disk, initializing [ 6047.408303] Lustre: Skipped 1 previous similar message [ 6047.410797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6047.414021] Lustre: Skipped 2 previous similar messages [ 6047.416778] Lustre: 213345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6048.513783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6048.517280] Lustre: Skipped 1 previous similar message [ 6048.519019] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6048.576801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6050.280093] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6055.322638] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.035643] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6058.146125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6060.362807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6060.456260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6064.097618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6064.100794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.101923] LustreError: Skipped 1 previous similar message [ 6064.103882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.118762] Lustre: Skipped 1 previous similar message [ 6064.129047] Lustre: Skipped 17 previous similar messages [ 6067.522831] Lustre: server umount lustre-OST0000 complete [ 6075.505132] Lustre: server umount lustre-MDT0000 complete [ 6077.310125] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364586 with bad export cookie 14672378455221948195 [ 6077.311241] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6077.313976] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6077.482995] Lustre: server umount lustre-MDT0001 complete [ 6079.209638] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:56:27 (1788364587) [ 6085.825753] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6089.590588] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6103.624412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.722495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6103.736715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6103.773333] Lustre: lustre-MDT0000: new disk, initializing [ 6103.807984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6105.579769] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6111.038616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.106975] Lustre: 217950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6111.226100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6113.035441] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6117.942320] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6120.389822] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6123.701465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.856133] Lustre: 219914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6125.431330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6126.469147] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6130.972372] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6132.829852] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6132.927873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6134.677439] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6134.756158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6135.777920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.779279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.779726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.779730] Lustre: Skipped 3 previous similar messages [ 6135.780754] LustreError: Skipped 2 previous similar messages [ 6135.787235] Lustre: Skipped 3 previous similar messages [ 6141.674270] Lustre: server umount lustre-OST0000 complete [ 6149.543197] Lustre: server umount lustre-MDT0000 complete [ 6151.136606] LustreError: 218943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.142352] LustreError: 218943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6151.338572] LustreError: 218743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364660 with bad export cookie 14672378455221949077 [ 6151.342227] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6151.343963] LustreError: 218743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6151.512227] Lustre: server umount lustre-MDT0001 complete [ 6155.604227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.939883] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6162.105977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.304699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6162.308912] Lustre: Skipped 6 previous similar messages [ 6164.141161] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6167.154145] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.283850] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6172.496182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6174.964202] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6177.840580] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.100682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.180785] Lustre: server umount lustre-OST0000 complete [ 6204.033728] Lustre: server umount lustre-MDT0000 complete [ 6206.168293] LustreError: 222862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364715 with bad export cookie 14672378455221949959 [ 6206.169678] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.180041] LustreError: 222862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6206.371066] Lustre: server umount lustre-MDT0001 complete [ 6313.968287] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6315.133932] Key type lgssc unregistered [ 6315.271376] LNet: 225806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6315.275897] LNetError: 225806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6315.286432] LNet: Removed LNI 192.168.201.125@tcp [ 6315.706265] Key type .llcrypt unregistered [ 6315.707834] Key type ._llcrypt unregistered [ 6326.063954] Key type ._llcrypt registered [ 6326.065518] Key type .llcrypt registered [ 6326.107133] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6333.861444] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6334.301640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6334.308914] alg: No test for adler32 (adler32-zlib) [ 6335.186408] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6335.303284] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6336.895166] Key type lgssc registered [ 6337.292417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.772446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6351.782463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.883098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6352.903611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6352.941363] Lustre: lustre-MDT0000: new disk, initializing [ 6352.975188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.986673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6354.965100] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6359.942479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.992787] Lustre: 229145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6360.021490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6360.024396] Lustre: Skipped 1 previous similar message [ 6360.063146] Lustre: lustre-MDT0001: new disk, initializing [ 6360.101854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6360.119095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6360.125780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6361.852954] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6366.932626] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6369.505737] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6372.435821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.542116] Lustre: lustre-OST0000: new disk, initializing [ 6372.544681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6372.547685] Lustre: 231111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6372.582125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.777392] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6375.546517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6375.550685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6375.589383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6379.030931] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6380.724920] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6380.807075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6382.394961] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6382.480372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6386.146126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.146736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.149024] LustreError: Skipped 1 previous similar message [ 6386.154747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6386.158174] Lustre: Skipped 1 previous similar message [ 6389.332373] Lustre: server umount lustre-OST0000 complete [ 6390.752854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6390.755864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.760587] Lustre: Skipped 1 previous similar message [ 6390.762794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.384960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.387059] Lustre: Skipped 3 previous similar messages [ 6396.827453] Lustre: server umount lustre-MDT0000 complete [ 6398.301447] LustreError: 229126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364907 with bad export cookie 10688231384501863162 [ 6398.305084] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.307772] LustreError: 229126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6398.443105] Lustre: server umount lustre-MDT0001 complete [ 6405.773340] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6409.870930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.033065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.616067] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6415.059356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.200621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6416.782270] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6417.900401] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6420.317881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.479905] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6425.831832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6425.902738] Lustre: lustre-OST0001: new disk, initializing [ 6425.905284] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6425.909212] Lustre: 236347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6425.947292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.949942] Lustre: Skipped 1 previous similar message [ 6428.148973] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6431.237888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6431.241132] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6431.269897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6438.312417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6446.187967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6450.571772] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:02:39 (1788364959) [ 6455.777464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.780845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6455.782273] Lustre: Skipped 4 previous similar messages [ 6455.783879] Lustre: Skipped 2 previous similar messages [ 6457.513561] Lustre: server umount lustre-MDT0000 complete [ 6458.921545] LustreError: 234022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364968 with bad export cookie 10688231384501864044 [ 6458.925457] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.926049] LustreError: 234022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6459.057820] Lustre: server umount lustre-MDT0001 complete [ 6470.095118] Lustre: server umount lustre-OST0000 complete [ 6481.768051] Lustre: server umount lustre-OST0001 complete [ 6484.077584] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6488.119378] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6502.152753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.242127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6502.253819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6502.290043] Lustre: lustre-MDT0000: new disk, initializing [ 6502.316707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.323435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6503.797710] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6508.358571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.408341] Lustre: 240289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6508.519250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6509.979526] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6513.861812] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6515.796594] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6518.196693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6518.296628] Lustre: lustre-OST0000: new disk, initializing [ 6518.298430] Lustre: Skipped 1 previous similar message [ 6518.299836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6518.302318] Lustre: Skipped 2 previous similar messages [ 6518.304671] Lustre: 242257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6519.488618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6519.491988] Lustre: Skipped 1 previous similar message [ 6519.493744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6519.517573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6520.402629] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6524.428942] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.926921] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6525.998947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6527.435767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6527.516249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6530.016546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.017674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.021196] LustreError: Skipped 1 previous similar message [ 6530.021922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.025840] Lustre: Skipped 1 previous similar message [ 6530.032023] Lustre: Skipped 2 previous similar messages [ 6534.290398] Lustre: server umount lustre-OST0000 complete [ 6539.231908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6539.234343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.238701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.792111] Lustre: server umount lustre-MDT0000 complete [ 6543.064167] LustreError: 241084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365052 with bad export cookie 10688231384501866165 [ 6543.066684] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.067273] LustreError: 241084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6543.197454] Lustre: server umount lustre-MDT0001 complete [ 6546.203268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.380718] LustreError: 244469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.387476] LustreError: 244469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6546.413348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6546.416445] Lustre: Skipped 2 previous similar messages [ 6547.784565] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6550.712224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.263120] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6554.812336] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6556.730752] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6559.159311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.377848] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6564.030990] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6708.703960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6708.707305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.714902] Lustre: Skipped 1 previous similar message [ 6708.717378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6708.719730] Lustre: Skipped 2 previous similar messages [ 6714.196282] Lustre: server umount lustre-OST0000 complete [ 6719.969358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6719.972600] LustreError: Skipped 1 previous similar message [ 6721.817342] Lustre: server umount lustre-MDT0000 complete [ 6723.141326] LustreError: 244496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365232 with bad export cookie 10688231384501867047 [ 6723.145061] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6723.145594] LustreError: 244496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6723.267726] Lustre: server umount lustre-MDT0001 complete [ 6829.962523] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6830.996806] Key type lgssc unregistered [ 6831.133388] LNet: 248076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6831.136659] LNetError: 248076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6831.146338] LNet: Removed LNI 192.168.201.125@tcp [ 6831.432145] Key type .llcrypt unregistered [ 6831.433863] Key type ._llcrypt unregistered [ 6839.951623] Key type ._llcrypt registered [ 6839.953148] Key type .llcrypt registered [ 6839.997170] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6846.585369] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6847.053243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6847.074980] alg: No test for adler32 (adler32-zlib) [ 6847.936537] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6848.030968] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6849.623116] Key type lgssc registered [ 6850.072443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6866.752415] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6870.466987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6870.478844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.571765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6871.585057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6871.622225] Lustre: lustre-MDT0000: new disk, initializing [ 6871.646764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.655477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6873.272402] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6878.023118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.069505] Lustre: 252117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.094283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6878.096942] Lustre: Skipped 1 previous similar message [ 6878.130488] Lustre: lustre-MDT0001: new disk, initializing [ 6878.167510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.181647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6878.185872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6879.725147] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6882.180227] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6884.815431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.922417] Lustre: lustre-OST0000: new disk, initializing [ 6884.924672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6884.928001] Lustre: 253749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6884.964325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.995154] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6891.023085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6891.027777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6891.066057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6891.485399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.554636] Lustre: lustre-OST0001: new disk, initializing [ 6891.556551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6891.559514] Lustre: 254620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6891.597383] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6893.660116] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6899.443624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6901.050713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6901.053652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6901.087776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6904.353630] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6908.895749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6908.898584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.903548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6910.198348] Lustre: server umount lustre-MDT0000 complete [ 6911.465118] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.800515] LustreError: 254619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365420 with bad export cookie 1700462839029327017 [ 6911.806950] LustreError: 254619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6911.809654] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.997133] Lustre: server umount lustre-MDT0001 complete [ 6925.919781] Lustre: server umount lustre-OST0000 complete [ 6940.429673] Lustre: server umount lustre-OST0001 complete [ 6953.023626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6962.628640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6973.168114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6986.264287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7002.680298] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7010.484913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.498204] Lustre: lustre-MDT0000: reset Object Index mappings [ 7010.822566] LustreError: 259619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.845471] LustreError: 259619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7010.876034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.656092] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7015.910747] LustreError: 259620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.585202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.597361] Lustre: lustre-MDT0001: reset Object Index mappings [ 7019.854220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7022.177696] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7023.703457] Lustre: 260763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7027.571622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7029.866863] LustreError: 261119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.838061] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7034.982800] LustreError: 261119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.996187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7035.000629] LustreError: 261119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7037.019898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.183482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7037.191438] Lustre: Skipped 1 previous similar message [ 7041.144630] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7042.548106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7046.599800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7048.736656] Lustre: 262631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7052.771972] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.777806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.791429] Lustre: Skipped 4 previous similar messages [ 7052.806048] Lustre: Skipped 3 previous similar messages [ 7057.865334] Lustre: server umount lustre-MDT0000 complete [ 7057.890907] LustreError: 259619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.897274] LustreError: 259619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7060.108673] LustreError: 259600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365569 with bad export cookie 1700462839029333695 [ 7060.109520] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.114780] LustreError: 259600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7060.309266] Lustre: server umount lustre-MDT0001 complete [ 7072.293173] Lustre: server umount lustre-OST0000 complete [ 7085.013484] Lustre: server umount lustre-OST0001 complete [ 7097.045175] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:13:25 (1788365605) [ 7106.984414] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 7113.103217] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7138.319485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.490156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7138.507933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7138.563743] Lustre: lustre-MDT0000: new disk, initializing [ 7138.611242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.623374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7141.128341] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7149.060295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.158458] Lustre: 266886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7149.206653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7149.209996] Lustre: Skipped 1 previous similar message [ 7149.261588] Lustre: lustre-MDT0001: new disk, initializing [ 7149.411974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7149.424591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7152.288422] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7158.334467] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7161.873345] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.135660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7166.327070] Lustre: lustre-OST0000: new disk, initializing [ 7166.333775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7166.339908] Lustre: 268852:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7167.464890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7167.468369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7167.571488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7170.691578] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7177.229434] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7180.360850] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7180.514757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7183.755939] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7183.893928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7187.940615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7187.949554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.950553] LustreError: Skipped 1 previous similar message [ 7187.951846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7187.971990] Lustre: Skipped 2 previous similar messages [ 7191.328964] Lustre: server umount lustre-OST0000 complete [ 7195.616385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7195.618704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.630103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7195.637985] Lustre: Skipped 1 previous similar message [ 7200.007300] Lustre: server umount lustre-MDT0000 complete [ 7202.318261] LustreError: 268851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365711 with bad export cookie 1700462839029335781 [ 7202.318905] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7202.323125] LustreError: 268851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7202.542402] Lustre: server umount lustre-MDT0001 complete [ 7216.123615] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7223.715054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.995767] LustreError: 271782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.028913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.032622] Lustre: Skipped 2 previous similar messages [ 7227.243251] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7233.528549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.704225] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7238.787638] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7243.688118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.874515] LustreError: 273268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.890226] LustreError: 273268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7248.411393] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7254.582321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.730166] Lustre: lustre-OST0001: new disk, initializing [ 7254.733803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7254.738678] Lustre: 274088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7256.800389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7256.811093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7256.889205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7259.038686] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7266.473692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7269.040877] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7273.243887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7273.274608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7279.810662] Lustre: server umount lustre-MDT0000 complete [ 7280.096803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.104493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.114289] Lustre: Skipped 1 previous similar message [ 7280.118090] LustreError: 272588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.751490] LustreError: 274850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365791 with bad export cookie 1700462839029336670 [ 7282.752724] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.758768] LustreError: 274850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7282.956368] Lustre: server umount lustre-MDT0001 complete [ 7296.064603] Lustre: server umount lustre-OST0000 complete [ 7308.727848] Lustre: server umount lustre-OST0001 complete [ 7320.349863] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7326.955819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.008516] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7335.261854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7338.211880] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7340.172164] Lustre: 278281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7344.917762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.236410] LustreError: 278635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.258239] LustreError: 278635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7349.747662] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7352.313699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7356.316791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7356.501416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7356.506019] Lustre: Skipped 6 previous similar messages [ 7361.153480] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7361.538570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7367.110126] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7369.614856] Lustre: 280156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7373.198670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7373.239909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7380.638661] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:18:08 (1788365888) [ 7396.834338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7396.840128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.856099] Lustre: Skipped 3 previous similar messages [ 7396.859881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7396.863045] Lustre: Skipped 2 previous similar messages [ 7400.470938] Lustre: server umount lustre-MDT0000 complete [ 7403.397361] LustreError: 277122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365912 with bad export cookie 1700462839029342984 [ 7403.403910] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7403.412201] LustreError: 277122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7407.597822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7407.601519] Lustre: Skipped 5 previous similar messages [ 7409.695737] Lustre: server umount lustre-MDT0001 complete [ 7418.790916] Lustre: server umount lustre-OST0000 complete [ 7427.507411] Lustre: server umount lustre-OST0001 complete [ 7432.311288] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 7439.860902] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7468.633184] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7474.420925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.553123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7474.574562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7474.629437] Lustre: lustre-MDT0000: new disk, initializing [ 7474.681411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7476.779258] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7483.353469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.432468] Lustre: 284409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7483.603240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7486.018533] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7492.836695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.000679] Lustre: 286040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7495.122587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7496.196071] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7502.455957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.544847] Lustre: 286913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7503.966746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7506.576745] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7513.206194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7515.488268] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7515.490853] Lustre: Skipped 1 previous similar message [ 7523.907925] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:20:32 (1788366032) [ 7529.442352] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.446962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7529.451925] Lustre: Skipped 6 previous similar messages [ 7529.456186] Lustre: Skipped 2 previous similar messages [ 7531.272646] Lustre: server umount lustre-MDT0000 complete [ 7533.289718] LustreError: 284403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366042 with bad export cookie 1700462839029350852 [ 7533.295885] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.296472] LustreError: 284403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7533.470883] Lustre: server umount lustre-MDT0001 complete [ 7545.822262] Lustre: server umount lustre-OST0000 complete [ 7557.080672] Lustre: server umount lustre-OST0001 complete [ 7562.041960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.304513] LustreError: 289160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.314306] LustreError: 289160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7565.563732] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7570.859269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.832331] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7578.605534] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7581.758175] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.654196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.293727] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7593.301343] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7606.713067] loop1: detected capacity change from 0 to 204800000 [ 7606.875288] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7606.880927] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7606.887445] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7610.548930] loop1: detected capacity change from 0 to 204800000 [ 7614.508324] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7614.583686] LustreError: 289150:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22387: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7614.593400] LustreError: 289150:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22387 [ 7614.597180] LustreError: 289150:0:(mgs_handler.c:517:mgs_target_reg()) fs22387-MDT0001: Error checking target index: rc = -13 [ 7614.603453] LustreError: fs22387-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7614.609955] LustreError: 292874:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7614.615364] LustreError: 292874:0:(tgt_mount.c:2129:server_put_super()) no obd fs22387-MDT0001 [ 7614.618976] LustreError: 292874:0:(tgt_mount.c:116:server_deregister_mount()) fs22387-MDT0001 not registered [ 7614.650496] Lustre: server umount fs22387-MDT0001 complete [ 7614.654727] LustreError: 292874:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7624.676119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.740220] Lustre: Setting parameter fs22387-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22387-MDT0001 [ 7624.782343] Lustre: srv-fs22387-MDT0001: No data found on store. Initialize space. [ 7624.785653] Lustre: Skipped 4 previous similar messages [ 7624.858408] Lustre: fs22387-MDT0001: new disk, initializing [ 7624.862553] Lustre: Skipped 3 previous similar messages [ 7624.961237] Lustre: fs22387-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.972042] Lustre: Skipped 7 previous similar messages [ 7624.990274] Lustre: srv-fs22387-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7624.998771] Lustre: 293341:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22387-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7625.009389] LustreError: 293341:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22387-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7629.511423] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7638.722215] Lustre: server umount fs22387-MDT0001 complete [ 7658.255430] loop1: detected capacity change from 0 to 204800000 [ 7664.999067] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.482403] Lustre: 294911:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22387-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7665.489750] LustreError: 294911:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22387-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7670.641526] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7673.335314] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7676.417677] Lustre: server umount fs22387-MDT0001 complete [ 7688.683593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.687843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.697241] Lustre: Skipped 4 previous similar messages [ 7688.705636] Lustre: Skipped 4 previous similar messages [ 7695.099866] Lustre: server umount lustre-MDT0000 complete [ 7703.519869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.608083] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.711729] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7709.175811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7716.354752] loop1: detected capacity change from 0 to 204800000 [ 7721.001948] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.299471] Lustre: 297359:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22387-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7721.312296] LustreError: 297359:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22387-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7725.084267] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7729.111657] Lustre: server umount fs22387-MDT0001 complete [ 7750.113362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7750.125196] LustreError: Skipped 3 previous similar messages [ 7755.239421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.244236] Lustre: Skipped 7 previous similar messages [ 7761.377515] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7761.597577] Lustre: server umount lustre-OST0000 complete [ 7770.838468] Lustre: server umount lustre-MDT0000 complete [ 7774.133468] LustreError: 289142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366283 with bad export cookie 1700462839029353533 [ 7774.142176] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7774.149566] LustreError: 289142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7774.413787] Lustre: server umount lustre-MDT0001 complete [ 7785.190574] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7787.539707] Key type lgssc unregistered [ 7787.825682] LNet: 299704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7787.829229] LNetError: 299704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7787.843553] LNet: Removed LNI 192.168.201.125@tcp [ 7788.637239] Key type .llcrypt unregistered [ 7788.639758] Key type ._llcrypt unregistered [ 7808.846710] Key type ._llcrypt registered [ 7808.848854] Key type .llcrypt registered [ 7808.973160] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7810.269825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.389550] alg: No test for adler32 (adler32-zlib) [ 7811.397885] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 7811.630965] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7813.271174] Key type lgssc registered [ 7814.332715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7821.357059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7821.375235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.711559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.946628] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7832.042202] LustreError: 300973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.062031] LustreError: 300973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7834.827765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.135285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.294814] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7847.275252] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7853.781708] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7860.296288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.589235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.631746] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.513507] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7866.878449] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.009791] LustreError: 302805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.044307] LustreError: 302805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7873.621730] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7878.118131] LustreError: 302805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.137676] LustreError: 302805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7888.360225] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.382269] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7889.642961] loop1: detected capacity change from 0 to 204800000 [ 7889.860571] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7889.868445] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7889.876085] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7894.697644] loop1: detected capacity change from 0 to 204800000 [ 7899.655086] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7899.818606] Lustre: fs19991-OST0000: new disk, initializing [ 7899.821675] Lustre: srv-fs19991-OST0000: No data found on store. Initialize space. [ 7899.830720] Lustre: 304709:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7899.921214] Lustre: fs19991-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7905.469213] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7906.313944] LustreError: 303484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.348418] LustreError: 303484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7913.398745] Lustre: server umount fs19991-OST0000 complete [ 7924.019480] loop1: detected capacity change from 0 to 204800000 [ 7928.658462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.816343] Lustre: fs19991-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.833311] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7939.572496] LustreError: 303484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.586185] LustreError: 303484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7943.113611] loop2: detected capacity change from 0 to 204800000 [ 7943.269953] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7943.277621] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7943.285104] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7944.699550] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7948.001298] loop2: detected capacity change from 0 to 204800000 [ 7949.815785] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7953.130234] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.207790] LustreError: fs19991-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7953.215899] LustreError: fs19991-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7953.220449] LustreError: 307748:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7953.225674] LustreError: 307748:0:(tgt_mount.c:2129:server_put_super()) no obd fs19991-OST0001 [ 7953.228390] LustreError: 307748:0:(tgt_mount.c:116:server_deregister_mount()) fs19991-OST0001 not registered [ 7953.242745] Lustre: server umount fs19991-OST0001 complete [ 7953.249700] LustreError: 307748:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7961.701394] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.815249] Lustre: fs19991-OST0001: new disk, initializing [ 7961.818458] Lustre: srv-fs19991-OST0001: No data found on store. Initialize space. [ 7961.823188] Lustre: 308299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7961.910386] Lustre: fs19991-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7967.461163] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7975.321875] Lustre: server umount fs19991-OST0000 complete [ 7983.743445] Lustre: server umount fs19991-OST0001 complete [ 7995.334035] loop1: detected capacity change from 0 to 204800000 [ 8000.228113] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8000.367229] Lustre: fs19991-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.588910] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.604860] LustreError: 302882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 8005.963457] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8012.520880] loop2: detected capacity change from 0 to 204800000 [ 8017.481570] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.940459] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8032.437664] Lustre: server umount fs19991-OST0000 complete [ 8041.776824] Lustre: server umount fs19991-OST0001 complete [ 8056.291626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.302890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.316781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.312409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.339589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.344487] Lustre: Skipped 1 previous similar message [ 8060.055741] Lustre: server umount lustre-OST0000 complete [ 8065.504462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8065.510239] LustreError: Skipped 1 previous similar message [ 8065.511659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.532240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.554206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.554801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.571432] Lustre: Skipped 1 previous similar message [ 8069.746926] Lustre: server umount lustre-MDT0000 complete [ 8072.879923] LustreError: 300955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366581 with bad export cookie 18416335717677777199 [ 8072.887221] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8072.905164] LustreError: 300955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8073.158548] Lustre: server umount lustre-MDT0001 complete [ 8083.178274] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8085.915898] Key type lgssc unregistered [ 8086.178678] LNet: 314218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.190997] LNetError: 314218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.237036] LNet: Removed LNI 192.168.201.125@tcp [ 8087.327160] Key type .llcrypt unregistered [ 8087.330096] Key type ._llcrypt unregistered [ 8119.708268] Key type ._llcrypt registered [ 8119.710831] Key type .llcrypt registered [ 8119.812175] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8134.593255] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8135.740282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8135.797762] alg: No test for adler32 (adler32-zlib) [ 8136.821452] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8137.039409] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8138.703359] Key type lgssc registered [ 8139.834816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8173.725336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8173.759754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.072676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8175.114274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8175.190603] Lustre: lustre-MDT0000: new disk, initializing [ 8175.288205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8175.310016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8179.379948] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8189.552032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.666511] Lustre: 318085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8189.733698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8189.737224] Lustre: Skipped 1 previous similar message [ 8189.819947] Lustre: lustre-MDT0001: new disk, initializing [ 8189.909863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.944125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8189.964257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8194.216954] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8203.391737] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8208.802647] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8215.830102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.233454] Lustre: lustre-OST0000: new disk, initializing [ 8216.241880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8216.257054] Lustre: 320053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8216.356841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.759484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8221.770587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8221.817584] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8221.903547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8230.590544] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8232.611807] Lustre: Mounted lustre-client [ 8233.461237] Lustre: Unmounted lustre-client [ 8242.663358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.664616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.672648] LustreError: Skipped 1 previous similar message [ 8242.673688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.703819] Lustre: Skipped 1 previous similar message [ 8248.325348] Lustre: server umount lustre-OST0000 complete [ 8253.417396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.425147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8253.432719] Lustre: Skipped 1 previous similar message [ 8253.449311] Lustre: Skipped 4 previous similar messages [ 8258.529072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8258.537049] Lustre: Skipped 1 previous similar message [ 8263.649252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8263.661872] Lustre: Skipped 1 previous similar message [ 8266.207115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8266.498474] Lustre: server umount lustre-MDT0000 complete [ 8270.549302] LustreError: 318880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366779 with bad export cookie 16493684542127518444 [ 8270.553613] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8270.555016] LustreError: 318880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8270.914307] Lustre: server umount lustre-MDT0001 complete [ 8279.888493] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8282.813462] Key type lgssc unregistered [ 8293.727877] LNet: 1 local NIs in recovery (showing 1): 192.168.201.125@tcp [ 8294.496581] LNet: 322213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8294.502131] LNetError: 322213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8361.073735] LNet: Removed LNI 192.168.201.125@tcp [ 8361.822468] Key type .llcrypt unregistered [ 8361.824745] Key type ._llcrypt unregistered [ 8377.098690] Key type ._llcrypt registered [ 8377.100914] Key type .llcrypt registered [ 8377.183423] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8390.686540] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8391.410790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8391.603040] alg: No test for adler32 (adler32-zlib) [ 8392.677088] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8392.852862] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8394.479250] Key type lgssc registered [ 8395.350395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8419.444893] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:35:27 (1788366927) [ 8424.834261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8424.865813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.161440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8426.191039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8426.269705] Lustre: lustre-MDT0000: new disk, initializing [ 8426.381126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.394389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8430.234315] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8439.075732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.177576] Lustre: 325434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.219822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8439.224544] Lustre: Skipped 1 previous similar message [ 8439.282356] Lustre: lustre-MDT0001: new disk, initializing [ 8439.360530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8439.415407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8439.423722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8443.037176] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8450.603864] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8455.103607] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8460.522288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8460.813674] Lustre: lustre-OST0000: new disk, initializing [ 8460.818492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8460.832775] Lustre: 327400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8460.939410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8466.661422] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8470.570606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8470.577133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8470.657388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8474.137038] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.860766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8485.861968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.869618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8485.873031] Lustre: Skipped 1 previous similar message [ 8485.885238] Lustre: Skipped 1 previous similar message [ 8490.977513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8490.983728] Lustre: Skipped 1 previous similar message [ 8491.852844] Lustre: server umount lustre-OST0000 complete [ 8495.591058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8495.598273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.609552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.703950] Lustre: server umount lustre-MDT0000 complete [ 8503.727275] LustreError: 325426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367012 with bad export cookie 18430896663989484503 [ 8503.730636] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.733489] LustreError: 325426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8503.984647] Lustre: server umount lustre-MDT0001 complete [ 8513.057844] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8515.076327] Key type lgssc unregistered [ 8515.327675] LNet: 329305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8515.332260] LNetError: 329305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8515.344771] LNet: Removed LNI 192.168.201.125@tcp [ 8515.981191] Key type .llcrypt unregistered [ 8515.983904] Key type ._llcrypt unregistered [ 8531.489542] Key type ._llcrypt registered [ 8531.491176] Key type .llcrypt registered [ 8531.553469] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8532.109273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8532.122458] alg: No test for adler32 (adler32-zlib) [ 8533.142241] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8533.311457] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8534.935190] Key type lgssc registered [ 8535.881139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8542.304511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8542.329273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.652879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.011808] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8552.937804] LustreError: 330558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.948182] LustreError: 330558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8553.208926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.482699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.904249] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8562.612520] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8566.969308] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8572.000874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.248055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8577.713989] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8586.932850] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.128374] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8602.330304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8609.252726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.255134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.286470] Lustre: Skipped 1 previous similar message [ 8609.307322] Lustre: Skipped 2 previous similar messages [ 8610.866063] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 8613.299562] Lustre: server umount lustre-MDT0000 complete [ 8614.377062] LustreError: 330553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.413268] LustreError: 330553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8615.958191] LustreError: 330555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8618.718425] LustreError: 331292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367127 with bad export cookie 11987543316985796917 [ 8618.733085] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8618.740676] LustreError: 331292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8619.526340] LustreError: 332400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8619.527295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8619.578432] Lustre: Skipped 1 previous similar message [ 8619.582180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8623.523413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8623.530986] Lustre: Skipped 1 previous similar message [ 8626.177887] LustreError: 330555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.204645] LustreError: 330555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8628.642615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8628.645647] Lustre: Skipped 1 previous similar message [ 8633.311190] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8633.933670] Lustre: server umount lustre-MDT0001 complete [ 8638.770381] Lustre: server umount lustre-OST0000 complete [ 8648.226203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.685557] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8657.212072] Lustre: 334741:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.125@tcp [ 8660.204478] Lustre: server umount MGS complete [ 8671.881071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.586965] LustreError: 335313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8672.624072] LustreError: 335313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8672.761676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.507094] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8688.527308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.787312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8693.083711] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8702.138660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.384387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.509484] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8718.038774] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8719.758929] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8725.792900] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8727.730772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8733.164333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8733.178756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.189845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8733.200701] Lustre: Skipped 1 previous similar message [ 8733.214468] Lustre: Skipped 3 previous similar messages [ 8734.628570] Lustre: server umount lustre-OST0000 complete [ 8739.808645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8739.814196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.119225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8753.634745] LustreError: 335314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.653151] LustreError: 335314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8753.709810] Lustre: server umount lustre-MDT0000 complete [ 8757.294929] LustreError: 335295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367266 with bad export cookie 11987543316985797757 [ 8757.304321] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.310717] LustreError: 335295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8757.630122] Lustre: server umount lustre-MDT0001 complete [ 8771.366354] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8775.074059] Key type lgssc unregistered [ 8775.462404] LNet: 338753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8775.472553] LNetError: 338753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8775.496333] LNet: Removed LNI 192.168.201.125@tcp [ 8776.664241] Key type .llcrypt unregistered [ 8776.668121] Key type ._llcrypt unregistered [ 8795.321884] Key type ._llcrypt registered [ 8795.324888] Key type .llcrypt registered [ 8795.433773] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8812.262933] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8814.047131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8814.091376] alg: No test for adler32 (adler32-zlib) [ 8815.282692] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8815.624567] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8817.327352] Key type lgssc registered [ 8818.658405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8856.945300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.995984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.333622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8858.370982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8858.502291] Lustre: lustre-MDT0000: new disk, initializing [ 8858.628604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.661393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8863.908295] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8874.833209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.981370] Lustre: 341919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8875.075029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8875.078549] Lustre: Skipped 1 previous similar message [ 8875.184858] Lustre: lustre-MDT0001: new disk, initializing [ 8875.356269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8875.417538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8875.444247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8880.237803] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8889.496585] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8895.059794] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.481464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8901.661530] Lustre: lustre-OST0000: new disk, initializing [ 8901.667585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8901.672637] Lustre: 343886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8901.780441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.934929] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8909.392885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8909.408468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8909.498731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8918.086082] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.867074] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8923.108574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8928.321809] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8928.513875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8935.397400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8935.400235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8935.403926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8935.411209] Lustre: Skipped 1 previous similar message [ 8935.431868] LustreError: Skipped 1 previous similar message [ 8936.736558] Lustre: server umount lustre-OST0000 complete [ 8942.048243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8942.054042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.088440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.092370] Lustre: Skipped 1 previous similar message [ 8945.632430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8945.637095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.589436] Lustre: server umount lustre-MDT0000 complete [ 8951.023142] LustreError: 345427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367460 with bad export cookie 5698238994120751301 [ 8951.039110] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8951.040143] LustreError: 345427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8951.427555] Lustre: server umount lustre-MDT0001 complete [ 8959.789138] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:44:27 (1788367467) [ 8977.232979] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8987.129850] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9019.560866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.748267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9019.767407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9019.820368] Lustre: lustre-MDT0000: new disk, initializing [ 9019.876706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.892171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9023.528140] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9031.784505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.907384] Lustre: 348643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9031.950192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9031.954812] Lustre: Skipped 1 previous similar message [ 9032.006920] Lustre: lustre-MDT0001: new disk, initializing [ 9032.083972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9032.119446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9032.134468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9035.100700] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9042.834888] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9047.473819] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9052.427751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9052.668053] Lustre: lustre-OST0000: new disk, initializing [ 9052.670905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9052.674132] Lustre: 350611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9052.739823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.990034] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9058.849386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9058.855584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9058.908337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9063.439582] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9084.390765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9084.391842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9084.402278] LustreError: Skipped 1 previous similar message [ 9084.404837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.414735] Lustre: Skipped 1 previous similar message [ 9084.417982] Lustre: Skipped 1 previous similar message [ 9088.335245] Lustre: server umount lustre-OST0000 complete [ 9093.612766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9093.620597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.636409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9093.638744] Lustre: Skipped 1 previous similar message [ 9097.354913] Lustre: server umount lustre-MDT0000 complete [ 9099.744975] LustreError: 348654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9099.766423] LustreError: 348654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9100.019323] LustreError: 348636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367609 with bad export cookie 5698238994120752190 [ 9100.020195] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.033065] LustreError: 348636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9100.214704] Lustre: server umount lustre-MDT0001 complete [ 9107.594640] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9109.626660] Key type lgssc unregistered [ 9109.831837] LNet: 352762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9109.839323] LNetError: 352762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9109.854075] LNet: Removed LNI 192.168.201.125@tcp [ 9110.485283] Key type .llcrypt unregistered [ 9110.488100] Key type ._llcrypt unregistered [ 9173.603556] Key type ._llcrypt registered [ 9173.605709] Key type .llcrypt registered [ 9173.701819] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9173.875553] Key type .llcrypt unregistered [ 9173.877546] Key type ._llcrypt unregistered [ 9229.593140] Key type ._llcrypt registered [ 9229.594502] Key type .llcrypt registered [ 9229.707894] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9229.931398] Key type .llcrypt unregistered [ 9229.933290] Key type ._llcrypt unregistered [ 9252.216796] Key type ._llcrypt registered [ 9252.218880] Key type .llcrypt registered [ 9252.293576] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9252.465525] Key type .llcrypt unregistered [ 9252.468724] Key type ._llcrypt unregistered [ 9273.903872] Key type ._llcrypt registered [ 9273.909215] Key type .llcrypt registered [ 9274.040893] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9274.267438] Key type .llcrypt unregistered [ 9274.273327] Key type ._llcrypt unregistered [ 9325.773466] Key type ._llcrypt registered [ 9325.774783] Key type .llcrypt registered [ 9325.842849] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9326.029415] Key type .llcrypt unregistered [ 9326.031502] Key type ._llcrypt unregistered [ 9347.812763] Key type ._llcrypt registered [ 9347.813791] Key type .llcrypt registered [ 9347.892608] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9348.098516] Key type .llcrypt unregistered [ 9348.101314] Key type ._llcrypt unregistered [ 9367.710638] Key type ._llcrypt registered [ 9367.711810] Key type .llcrypt registered [ 9367.779854] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9367.942486] Key type .llcrypt unregistered [ 9367.943589] Key type ._llcrypt unregistered [ 9387.694403] Key type ._llcrypt registered [ 9387.695708] Key type .llcrypt registered [ 9387.757933] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9387.931515] Key type .llcrypt unregistered [ 9387.933323] Key type ._llcrypt unregistered [ 9407.917797] Key type ._llcrypt registered [ 9407.919737] Key type .llcrypt registered [ 9408.007382] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9408.197422] Key type .llcrypt unregistered [ 9408.199338] Key type ._llcrypt unregistered [ 9428.547184] Key type ._llcrypt registered [ 9428.548561] Key type .llcrypt registered [ 9428.624641] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9428.793319] Key type .llcrypt unregistered [ 9428.794997] Key type ._llcrypt unregistered [ 9468.726143] Key type ._llcrypt registered [ 9468.729298] Key type .llcrypt registered [ 9468.820081] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9469.046636] Key type .llcrypt unregistered [ 9469.050667] Key type ._llcrypt unregistered [ 9488.363525] Key type ._llcrypt registered [ 9488.366240] Key type .llcrypt registered [ 9488.440396] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9488.628296] Key type .llcrypt unregistered [ 9488.635146] Key type ._llcrypt unregistered [ 9509.037382] Key type ._llcrypt registered [ 9509.041414] Key type .llcrypt registered [ 9509.104322] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9509.280172] Key type .llcrypt unregistered [ 9509.281888] Key type ._llcrypt unregistered [ 9546.420813] Key type ._llcrypt registered [ 9546.422783] Key type .llcrypt registered [ 9546.505320] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9558.330052] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9559.193067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9559.210594] alg: No test for adler32 (adler32-zlib) [ 9560.232716] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 9560.427424] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9562.071207] Key type lgssc registered [ 9562.941740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9584.282424] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9584.295608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.443502] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9585.463245] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9585.511268] Lustre: tst250fs-MDT0000: new disk, initializing [ 9585.570434] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.585951] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9588.085268] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9594.745823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.819532] Lustre: 372701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9594.863830] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9594.866406] Lustre: Skipped 1 previous similar message [ 9594.906936] Lustre: tst250fs-MDT0001: new disk, initializing [ 9594.969278] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9595.000034] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9595.010500] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9597.747673] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9600.980382] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9605.069707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9605.265543] Lustre: tst250fs-OST0000: new disk, initializing [ 9605.269343] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9605.273644] Lustre: 374330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9605.331203] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9606.556136] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9606.563936] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9606.651843] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9609.119493] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9615.781417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9615.881359] Lustre: tst250fs-OST0001: new disk, initializing [ 9615.884050] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9615.887831] Lustre: 375204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9615.946899] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9619.745804] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9623.093539] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9623.107396] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9623.181104] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9623.954390] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9638.890324] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9638.893030] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9638.896328] Lustre: Skipped 1 previous similar message [ 9638.911407] Lustre: Skipped 3 previous similar messages [ 9644.007020] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9644.011086] Lustre: Skipped 3 previous similar messages [ 9649.122234] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9649.129481] Lustre: Skipped 2 previous similar messages [ 9653.219068] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9653.338907] Lustre: server umount tst250fs-MDT0000 complete [ 9655.605333] LustreError: 372692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368164 with bad export cookie 393544404678237909 [ 9655.614798] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9655.616974] LustreError: 372692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9655.880065] Lustre: server umount tst250fs-MDT0001 complete [ 9668.642057] Lustre: server umount tst250fs-OST0000 complete [ 9680.203550] Lustre: server umount tst250fs-OST0001 complete [ 9691.713940] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9696.587403] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9715.582925] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:57:03 (1788368223) [ 9716.444740] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9726.969572] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9733.451288] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9755.107733] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9519 sec ========== 12:57:43 (1788368263) [ 9756.452223] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:57:44 (1788368264) === [ 9759.006583] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:57:46 (1788368266) === [ 9786.814582] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9788.912498] Key type lgssc unregistered [ 9789.145489] LNet: 384323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9789.150807] LNetError: 384323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9789.161775] LNet: Removed LNI 192.168.201.125@tcp [ 9789.758162] Key type .llcrypt unregistered [ 9789.760770] Key type ._llcrypt unregistered