[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 582729924 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005022] kvm-guest: setup PV IPIs [ 0.007909] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010184] LSM: Security Framework initializing [ 0.011089] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.021698] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026777] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031178] *** VALIDATE tmpfs *** [ 0.033195] *** VALIDATE proc *** [ 0.034297] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043301] debug: unmapping init [mem 0xffffffff9a059000-0xffffffff9a060fff] [ 0.046162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.048976] ... bit width: 48 [ 0.049023] ... generic registers: 4 [ 0.050020] ... value mask: 0000ffffffffffff [ 0.051025] ... max period: 00007fffffffffff [ 0.052036] ... fixed-purpose events: 3 [ 0.053025] ... event mask: 000000070000000f [ 0.054490] rcu: Hierarchical SRCU implementation. [ 0.057125] smp: Bringing up secondary CPUs ... [ 0.058909] x86: Booting SMP configuration: [ 0.059058] .... node #0, CPUs: #1 #2 #3 [ 0.065050] smp: Brought up 1 node, 4 CPUs [ 0.067024] smpboot: Max logical packages: 1 [ 0.068017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152637] node 0 deferred pages initialised in 82ms [ 0.155137] devtmpfs: initialized [ 0.156481] x86/mm: Memory block size: 128MB [ 0.160160] gcov: version magic: 0x41383552 [ 0.163366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168495] pinctrl core: initialized pinctrl subsystem [ 0.170244] [ 0.170642] ************************************************************* [ 0.173013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175014] ** ** [ 0.177014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179014] ** ** [ 0.182020] ** This means that this kernel is built to expose internal ** [ 0.184014] ** IOMMU data structures, which may compromise security on ** [ 0.187017] ** your system. ** [ 0.189012] ** ** [ 0.191020] ** If you see this message and you are not debugging the ** [ 0.194017] ** kernel, report this immediately to your vendor! ** [ 0.196013] ** ** [ 0.198017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200016] ************************************************************* [ 0.202692] NET: Registered protocol family 16 [ 0.205426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213110] cpuidle: using governor menu [ 0.215020] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218303] PCI: Using configuration type 1 for base access [ 0.220151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229306] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.232046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236397] cryptd: max_cpu_qlen set to 1000 [ 0.239099] ACPI: Added _OSI(Module Device) [ 0.240009] ACPI: Added _OSI(Processor Device) [ 0.241010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249011] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255133] ACPI: Interpreter enabled [ 0.256077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258021] ACPI: Using IOAPIC for interrupt routing [ 0.260113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273873] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286329] acpiphp: Slot [2] registered [ 0.287088] acpiphp: Slot [5] registered [ 0.288027] acpiphp: Slot [6] registered [ 0.290099] acpiphp: Slot [7] registered [ 0.291108] acpiphp: Slot [8] registered [ 0.292095] acpiphp: Slot [9] registered [ 0.292992] acpiphp: Slot [10] registered [ 0.294061] acpiphp: Slot [3] registered [ 0.294769] acpiphp: Slot [4] registered [ 0.296062] acpiphp: Slot [11] registered [ 0.297104] acpiphp: Slot [12] registered [ 0.299205] acpiphp: Slot [13] registered [ 0.300159] acpiphp: Slot [14] registered [ 0.302085] acpiphp: Slot [15] registered [ 0.303113] acpiphp: Slot [16] registered [ 0.304115] acpiphp: Slot [17] registered [ 0.306106] acpiphp: Slot [18] registered [ 0.307085] acpiphp: Slot [19] registered [ 0.308089] acpiphp: Slot [20] registered [ 0.310116] acpiphp: Slot [21] registered [ 0.311095] acpiphp: Slot [22] registered [ 0.313086] acpiphp: Slot [23] registered [ 0.314096] acpiphp: Slot [24] registered [ 0.315100] acpiphp: Slot [25] registered [ 0.317074] acpiphp: Slot [26] registered [ 0.318133] acpiphp: Slot [27] registered [ 0.319122] acpiphp: Slot [28] registered [ 0.321117] acpiphp: Slot [29] registered [ 0.322094] acpiphp: Slot [30] registered [ 0.324086] acpiphp: Slot [31] registered [ 0.325080] PCI host bridge to bus 0000:00 [ 0.326037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337756] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339894] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346970] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351929] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363704] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365621] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370942] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398729] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415032] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470404] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.500037] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.546134] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.563022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.574031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.600022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.611573] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.622027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.631019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.659155] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.673028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.693021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.704781] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.710016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.718018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.735035] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.745355] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.748444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.750356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.754409] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.757259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763126] iommu: Default domain type: Passthrough [ 0.766530] SCSI subsystem initialized [ 0.768184] ACPI: bus type USB registered [ 0.770170] usbcore: registered new interface driver usbfs [ 0.772096] usbcore: registered new interface driver hub [ 0.774098] usbcore: registered new device driver usb [ 0.776184] pps_core: LinuxPPS API ver. 1 registered [ 0.779014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782064] PTP clock support registered [ 0.785064] EDAC MC: Ver: 3.0.0 [ 0.787142] PCI: Using ACPI for IRQ routing [ 0.788488] NetLabel: Initializing [ 0.788902] NetLabel: domain hash size = 128 [ 0.791014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.793109] NetLabel: unlabeled traffic allowed by default [ 0.794152] vgaarb: loaded [ 0.796290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.798016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.804339] clocksource: Switched to clocksource kvm-clock [ 0.905817] VFS: Disk quotas dquot_6.6.0 [ 0.907430] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.910030] *** VALIDATE ramfs *** [ 0.911284] *** VALIDATE hugetlbfs *** [ 0.912876] pnp: PnP ACPI init [ 0.915243] pnp: PnP ACPI: found 6 devices [ 0.945542] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.949578] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.952701] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.955297] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.958052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.960828] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.964041] NET: Registered protocol family 2 [ 0.967270] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.972730] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.975637] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.981985] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.984697] TCP: Hash tables configured (established 65536 bind 65536) [ 0.986793] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.989028] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991149] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993409] NET: Registered protocol family 1 [ 0.995717] RPC: Registered named UNIX socket transport module. [ 0.997504] RPC: Registered udp transport module. [ 0.999351] RPC: Registered tcp transport module. [ 1.001333] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.003806] NET: Registered protocol family 44 [ 1.005702] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.008202] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.010408] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.012292] PCI: CLS 0 bytes, default 64 [ 1.013609] Unpacking initramfs... [ 2.409863] debug: unmapping init [mem 0xffffa052fcc54000-0xffffa052fffbffff] [ 2.413650] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.415905] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.418524] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.913645] Initialise system trusted keyrings [ 2.915329] Key type blacklist registered [ 2.917222] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.926321] zbud: loaded [ 2.930394] *** VALIDATE nfs *** [ 2.931546] *** VALIDATE nfs4 *** [ 2.933202] pstore: using deflate compression [ 2.936405] Platform Keyring initialized [ 3.045252] NET: Registered protocol family 38 [ 3.047104] Key type asymmetric registered [ 3.048475] Asymmetric key parser 'x509' registered [ 3.050325] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.052986] io scheduler mq-deadline registered [ 3.054738] io scheduler kyber registered [ 3.056329] io scheduler bfq registered [ 3.058479] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.061124] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.064238] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.067389] ACPI: Power Button [PWRF] [ 3.073714] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.080324] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.096383] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.104296] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.121644] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.148798] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.178319] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.183185] Non-volatile memory driver v1.3 [ 3.184283] Linux agpgart interface v0.103 [ 3.220790] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.223465] vda: detected capacity change from 0 to 74711040 [ 3.239659] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.243144] vdb: detected capacity change from 0 to 1073741824 [ 3.258781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.260860] vdc: detected capacity change from 0 to 2621440000 [ 3.301647] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305206] vdd: detected capacity change from 0 to 2621440000 [ 3.326526] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.329574] vde: detected capacity change from 0 to 4294967296 [ 3.348628] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351715] vdf: detected capacity change from 0 to 4294967296 [ 3.370221] libphy: Fixed MDIO Bus: probed [ 3.383139] usbcore: registered new interface driver usbserial_generic [ 3.387075] usbserial: USB Serial support registered for generic [ 3.390732] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.397254] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.399016] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.401637] mousedev: PS/2 mouse device common for all mice [ 3.405045] rtc_cmos 00:05: RTC can wake from S4 [ 3.406428] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.409356] rtc_cmos 00:05: registered as rtc0 [ 3.411190] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.414233] intel_pstate: CPU model not supported [ 3.414399] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.419707] hid: raw HID events driver (C) Jiri Kosina [ 3.420736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.421970] usbcore: registered new interface driver usbhid [ 3.427174] usbhid: USB HID core driver [ 3.428633] drop_monitor: Initializing network drop monitor service [ 3.430781] Initializing XFRM netlink socket [ 3.432724] NET: Registered protocol family 10 [ 3.435633] Segment Routing with IPv6 [ 3.436656] NET: Registered protocol family 17 [ 3.438382] mpls_gso: MPLS GSO support [ 3.443733] RAS: Correctable Errors collector initialized. [ 3.446044] AVX version of gcm_enc/dec engaged. [ 3.447815] AES CTR mode by8 optimization enabled [ 3.517586] sched_clock: Marking stable (3517560826, 0)->(4445535553, -927974727) [ 3.520746] registered taskstats version 1 [ 3.522648] Loading compiled-in X.509 certificates [ 3.524755] zswap: loaded using pool lzo/zbud [ 3.548452] Key type big_key registered [ 3.559951] Key type encrypted registered [ 3.561183] ima: No TPM chip found, activating TPM-bypass! [ 3.563326] ima: Allocated hash algorithm: sha1 [ 3.564819] ima: No architecture policies found [ 3.566479] evm: Initialising EVM extended attributes: [ 3.567829] evm: security.selinux [ 3.568513] evm: security.ima [ 3.569162] evm: security.capability [ 3.569930] evm: HMAC attrs: 0x1 [ 3.571728] rtc_cmos 00:05: setting system clock to 2026-08-19 06:03:13 UTC (1787119393) [ 3.576880] debug: unmapping init [mem 0xffffffff9b003000-0xffffffff9b1fffff] [ 3.579521] debug: unmapping init [mem 0xffffffff99d82000-0xffffffff9a058fff] [ 3.589099] Write protecting the kernel read-only data: 28672k [ 3.591991] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 3.594375] debug: unmapping init [mem 0xffffffff98d14000-0xffffffff98dfffff] [ 3.626896] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.635213] systemd[1]: Detected virtualization kvm. [ 3.637507] systemd[1]: Detected architecture x86-64. [ 3.640277] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.665485] systemd[1]: No hostname configured. [ 3.666755] systemd[1]: Set hostname to . [ 3.668324] random: systemd: uninitialized urandom read (16 bytes read) [ 3.670423] systemd[1]: Initializing machine ID from random generator. [ 3.783181] random: systemd: uninitialized urandom read (16 bytes read) [ 3.785696] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.789296] random: systemd: uninitialized urandom read (16 bytes read) [ 3.791786] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.795817] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.335160] device-mapper: uevent: version 1.0.3 [ 4.336581] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.948689] virtio_net virtio0 ens2: renamed from eth0 [ 4.988085] random: fast init done [ 5.056301] scsi host0: ata_piix [ 5.105265] scsi host1: ata_piix [ 5.106860] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.109291] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.643718] dracut-initqueue[575]: RTNETLINK answers: File exists [ 10.078128] random: crng init done [ 10.079532] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.354197] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.464520] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.658576] SELinux: Disabled at runtime. [ 11.717693] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.725949] systemd[1]: Detected virtualization kvm. [ 11.727643] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.251767] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.254501] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.259624] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.265317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.268672] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.275248] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.283381] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 12.339382] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.713650] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.022000] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.045326] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.168252] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.179811] EDAC sbridge: Ver: 1.1.2 [ 14.944563] Key type dns_resolver registered [ 15.278515] NFS: Registering the id_resolver key type [ 15.280584] Key type id_resolver registered [ 15.282335] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg207-server login: [ 44.596159] hrtimer: interrupt took 4124936 ns [ 47.412793] libcfs: loading out-of-tree module taints kernel. [ 47.450989] Key type ._llcrypt registered [ 47.452766] Key type .llcrypt registered [ 47.535388] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 63.395511] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 64.986456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.006896] alg: No test for adler32 (adler32-zlib) [ 66.784262] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 67.570389] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 69.311142] Key type lgssc registered [ 71.024151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.556157] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.729451] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 142.069450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.138417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.476606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.534774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.662696] Lustre: lustre-MDT0000: new disk, initializing [ 143.828423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.875132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.465927] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.589498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.747650] Lustre: 6499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.795910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.806435] Lustre: Skipped 1 previous similar message [ 161.897288] Lustre: lustre-MDT0001: new disk, initializing [ 161.973261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.025535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.040703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.180495] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.625670] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.106168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.351697] Lustre: lustre-OST0000: new disk, initializing [ 178.354829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.360547] Lustre: 8433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.433203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.700812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.713525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.791167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.434501] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.817106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.967639] Lustre: lustre-OST0001: new disk, initializing [ 197.970761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.974794] Lustre: 9505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.038235] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.759303] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.410460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.418427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.463450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.452582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.695437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.175294] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.908957] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing yml_node [ 241.324573] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 243.932629] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 246.666664] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 248.156979] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 02:07:16 EDT 2026 [ 266.351157] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 268.160436] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.151805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.159177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.172667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.738521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.742996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.766631] Lustre: Skipped 2 previous similar messages [ 288.782060] Lustre: Skipped 3 previous similar messages [ 290.741127] Lustre: server umount lustre-MDT0000 complete [ 294.287655] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119684 with bad export cookie 18196887259977996888 [ 294.299754] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.300323] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 298.979383] LustreError: 10083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.981478] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.989382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.012606] LustreError: 10083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 299.043549] Lustre: Skipped 1 previous similar message [ 300.000506] LustreError: 6504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.025861] LustreError: 6504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 300.845822] Lustre: server umount lustre-MDT0001 complete [ 305.278500] Lustre: server umount lustre-OST0000 complete [ 309.819437] Lustre: server umount lustre-OST0001 complete [ 317.498575] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 326.300497] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 360.250268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.537250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.589307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.726481] Lustre: lustre-MDT0000: new disk, initializing [ 360.869036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.900349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 365.259617] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 376.048164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.220941] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.370244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.377580] Lustre: Skipped 1 previous similar message [ 376.528185] Lustre: lustre-MDT0001: new disk, initializing [ 376.638727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.683109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 376.698736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 381.559206] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 391.192305] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.126587] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 404.225132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 404.498500] Lustre: lustre-OST0000: new disk, initializing [ 404.505824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 404.519988] Lustre: 17031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 404.635274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.271205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 406.286148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 406.466534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 411.355935] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 420.943432] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 425.697059] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.104766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.216108] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.425384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.217528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.224997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.232461] LustreError: Skipped 1 previous similar message [ 437.237172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.255842] Lustre: Skipped 1 previous similar message [ 437.276550] Lustre: Skipped 4 previous similar messages [ 438.797372] Lustre: server umount lustre-OST0000 complete [ 443.360873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 443.384512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.410744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.926555] Lustre: server umount lustre-MDT0000 complete [ 452.589736] LustreError: 15073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.608624] LustreError: 15073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 452.684713] LustreError: 17030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119842 with bad export cookie 18196887259977998421 [ 452.685984] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.700311] LustreError: 17030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 453.087926] Lustre: server umount lustre-MDT0001 complete [ 459.148474] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:10:46 (1787119846) [ 479.377792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.872920] LustreError: 19905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.977553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.004629] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 485.350075] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.138818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.125524] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 507.126216] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.692283] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.993896] Lustre: Failing over lustre-MDT0000 [ 516.254952] Lustre: server umount lustre-MDT0000 complete [ 520.098924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.109477] LustreError: 20633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.117541] Lustre: Skipped 2 previous similar messages [ 520.144142] LustreError: 20633:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 535.096737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.315721] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.644689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 540.647050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 540.670212] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 540.955433] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 550.555759] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 556.001253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.006551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.010490] Lustre: Skipped 1 previous similar message [ 556.012496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.017996] Lustre: Skipped 3 previous similar messages [ 560.774354] Lustre: server umount lustre-MDT0000 complete [ 561.123105] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.151552] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 564.944412] LustreError: 19886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119954 with bad export cookie 18196887259977999835 [ 564.975400] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.336769] Lustre: server umount lustre-MDT0001 complete [ 574.706290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.572702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.578472] Lustre: Skipped 2 previous similar messages [ 580.083978] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 588.540097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.146821] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 602.162329] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.812041] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.921552] Lustre: Failing over lustre-MDT0000 [ 610.650861] Lustre: server umount lustre-MDT0000 complete [ 610.787892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.808853] LustreError: 23236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.833488] LustreError: 23236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 626.982386] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120000/real 1787120000] req@ffffa0524368ea00 x1873930571930496/t0(0) o400->MGC192.168.202.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787120016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 627.030561] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.181315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.387635] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfc88548d3cd1e985 [ 636.401847] Lustre: MGC192.168.202.107@tcp: Connection restored to 0@lo (at 0@lo) [ 636.421484] Lustre: Skipped 1 previous similar message [ 636.822948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 640.634535] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 642.036302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 642.042339] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 642.071658] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 649.318439] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 653.281398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 659.546422] Lustre: server umount lustre-MDT0000 complete [ 663.035831] LustreError: 23222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120052 with bad export cookie 18196887259978000773 [ 663.042852] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.050025] LustreError: 23222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 663.267428] Lustre: server umount lustre-MDT0001 complete [ 673.175475] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:14:20 (1787120060) [ 692.602493] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 704.794995] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 745.290938] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 757.461193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.809893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 757.847374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 757.952947] Lustre: lustre-MDT0000: new disk, initializing [ 758.065935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.071198] Lustre: Skipped 2 previous similar messages [ 758.088690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 763.679873] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 774.625808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.719722] Lustre: 29617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 774.981791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 780.831075] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 791.213580] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 799.070611] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 805.354159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.496806] Lustre: 31584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 806.677488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 806.713602] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 806.796509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 806.815946] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 812.228477] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 821.161375] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 826.496979] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 847.843104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.845909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.853412] Lustre: Skipped 4 previous similar messages [ 847.854340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 847.854344] Lustre: Skipped 5 previous similar messages [ 851.483876] Lustre: server umount lustre-OST0000 complete [ 857.056577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.062718] LustreError: Skipped 1 previous similar message [ 861.850650] Lustre: server umount lustre-MDT0000 complete [ 863.201126] LustreError: 30492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.208775] LustreError: 30492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 865.281173] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120255 with bad export cookie 18196887259978001186 [ 865.284452] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.289012] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 865.768810] Lustre: server umount lustre-MDT0001 complete [ 876.995498] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 879.624339] Key type lgssc unregistered [ 879.889929] LNet: 33564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.907097] LNetError: 33564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.926464] LNet: Removed LNI 192.168.202.107@tcp [ 881.243140] Key type .llcrypt unregistered [ 881.247132] Key type ._llcrypt unregistered [ 903.542720] Key type ._llcrypt registered [ 903.545880] Key type .llcrypt registered [ 903.663391] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 904.664429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.704075] alg: No test for adler32 (adler32-zlib) [ 905.767508] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 906.082262] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 907.768719] Key type lgssc registered [ 909.186524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.064968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.085219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.593066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.267440] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 929.766255] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.782831] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 934.572820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.892498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.206284] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 948.276606] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.629688] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.865428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.213312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.354902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 966.227152] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 972.263459] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 979.697845] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 987.163775] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:19:35 (1787120375) [ 988.676547] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 990.332617] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 995.898426] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:19:44 (1787120384) [ 1002.832230] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:19:50 (1787120390) [ 1003.782646] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1010.508446] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1010.513669] Lustre: Skipped 2 previous similar messages [ 1011.896244] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:20:00 (1787120400) [ 1015.018710] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1015.024134] Lustre: Skipped 3 previous similar messages [ 1030.617903] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1030.622299] Lustre: Skipped 6 previous similar messages [ 1032.846430] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:20:20 (1787120420) [ 1035.394108] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1038.620657] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1045.843379] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:20:33 (1787120433) [ 1047.558342] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1047.563247] Lustre: Skipped 3 previous similar messages [ 1059.022244] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:20:47 (1787120447) [ 1072.097083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1072.100117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.107291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.132942] Lustre: Skipped 1 previous similar message [ 1077.224964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.236309] Lustre: Skipped 2 previous similar messages [ 1077.800381] Lustre: server umount lustre-OST0000 complete [ 1082.336192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.360052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.370525] Lustre: Skipped 1 previous similar message [ 1087.456574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.464296] Lustre: Skipped 1 previous similar message [ 1087.568342] Lustre: server umount lustre-MDT0000 complete [ 1091.075710] LustreError: 35715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120481 with bad export cookie 10273181210132350145 [ 1091.078239] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1091.088072] LustreError: 35715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1091.714314] Lustre: server umount lustre-MDT0001 complete [ 1102.077563] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1104.653382] Key type lgssc unregistered [ 1104.948831] LNet: 42832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.956491] LNetError: 42832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.979966] LNet: Removed LNI 192.168.202.107@tcp [ 1105.600335] Key type .llcrypt unregistered [ 1105.605572] Key type ._llcrypt unregistered [ 1122.186498] Key type ._llcrypt registered [ 1122.188723] Key type .llcrypt registered [ 1122.290399] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 1138.055615] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1139.724219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.764865] alg: No test for adler32 (adler32-zlib) [ 1140.788423] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1141.074609] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1142.727647] Key type lgssc registered [ 1143.787693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.567249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.591074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.937046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1180.997418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1181.120423] Lustre: lustre-MDT0000: new disk, initializing [ 1181.258588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.287294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1186.150131] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1196.314496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.474423] Lustre: 45986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1196.545409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1196.548293] Lustre: Skipped 1 previous similar message [ 1196.605564] Lustre: lustre-MDT0001: new disk, initializing [ 1196.732227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.794806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1196.814467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1201.728572] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1210.846493] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.512088] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.470614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.745328] Lustre: lustre-OST0000: new disk, initializing [ 1221.751867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1221.762285] Lustre: 47953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1221.886952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.960075] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1229.869052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1229.876400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1229.971923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1234.815902] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1239.519971] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1239.717529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1244.551058] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1244.818849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1250.278242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.289875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.302551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.792055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.815905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.852076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.909463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.922851] Lustre: Skipped 1 previous similar message [ 1261.023182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1261.029124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.058717] Lustre: Skipped 1 previous similar message [ 1261.272269] Lustre: server umount lustre-OST0000 complete [ 1266.147494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.153830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.161050] Lustre: Skipped 1 previous similar message [ 1266.181767] Lustre: Skipped 1 previous similar message [ 1270.977229] Lustre: server umount lustre-MDT0000 complete [ 1274.870509] LustreError: 45979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120664 with bad export cookie 14652882471861368657 [ 1274.872343] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.881908] LustreError: 45979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1275.156916] Lustre: server umount lustre-MDT0001 complete [ 1285.804742] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:24:33 (1787120673) [ 1292.997898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.345645] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.362742] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1293.463291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.659256] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1298.920162] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.039815] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.233147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.512496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1312.977549] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1320.396934] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1326.351718] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1331.986516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.276538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.711739] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1344.947470] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1349.640375] LustreError: 52945:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1350.488918] Lustre: Modifying parameter general.timeout=1 in log params [ 1352.350562] Lustre: Modifying parameter general.timeout=3 in log params [ 1352.355042] Lustre: Skipped 1 previous similar message [ 1354.991779] Lustre: Modifying parameter general.timeout=6 in log params [ 1354.996766] Lustre: Skipped 2 previous similar messages [ 1359.790571] Lustre: Modifying parameter general.timeout=12 in log params [ 1359.794214] Lustre: Skipped 5 previous similar messages [ 1367.794749] Lustre: Modifying parameter general.timeout=21 in log params [ 1367.800745] Lustre: Skipped 8 previous similar messages [ 1383.978632] Lustre: Modifying parameter general.timeout=41 in log params [ 1383.981981] Lustre: Skipped 19 previous similar messages [ 1416.215960] Lustre: Modifying parameter general.timeout=80 in log params [ 1416.219138] Lustre: Skipped 38 previous similar messages [ 1469.575986] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:27:37 (1787120857) [ 1472.158225] Lustre: 59883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1473.839647] Lustre: 59982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1477.791653] Lustre: 60179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1477.798385] Lustre: 60179:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1483.355190] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1483.360203] Lustre: Skipped 57 previous similar messages [ 1507.330329] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1507.339953] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1507.352542] LustreError: 61750:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1511.874591] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:28:19 (1787120899) [ 1523.681185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.688695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.700022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.337166] LustreError: 51144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.365171] Lustre: server umount lustre-MDT0000 complete [ 1530.377691] LustreError: 51144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1533.893592] LustreError: 50396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120923 with bad export cookie 14652882471861369539 [ 1533.907267] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1533.908343] LustreError: 50396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1534.398462] Lustre: server umount lustre-MDT0001 complete [ 1548.570874] Lustre: server umount lustre-OST0000 complete [ 1578.242072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.804664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1579.266968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.459243] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1592.971181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.057611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1593.085559] Lustre: 64283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1598.832360] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1606.114641] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1612.641889] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1619.750376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1619.913093] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1620.188818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.194484] Lustre: Skipped 1 previous similar message [ 1626.593848] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1634.336995] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1636.595664] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1636.604395] Lustre: Skipped 15 previous similar messages [ 1636.641497] Lustre: 66848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1645.026976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.035505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.046117] Lustre: Skipped 2 previous similar messages [ 1645.055759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.060489] Lustre: Skipped 3 previous similar messages [ 1650.710687] Lustre: server umount lustre-OST0000 complete [ 1655.778248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.803546] Lustre: Skipped 2 previous similar messages [ 1661.485994] Lustre: server umount lustre-MDT0000 complete [ 1666.020519] LustreError: 64294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.061294] LustreError: 64294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1667.383239] LustreError: 64276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121057 with bad export cookie 14652882471861372493 [ 1667.404720] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.405657] LustreError: 64276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1667.720144] Lustre: server umount lustre-MDT0001 complete [ 1680.146648] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1682.839244] Key type lgssc unregistered [ 1683.186346] LNet: 68063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.213047] LNetError: 68063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.263563] LNet: Removed LNI 192.168.202.107@tcp [ 1685.318948] Key type .llcrypt unregistered [ 1685.322817] Key type ._llcrypt unregistered [ 1710.616942] Key type ._llcrypt registered [ 1710.625134] Key type .llcrypt registered [ 1710.771574] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1711.708706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.742984] alg: No test for adler32 (adler32-zlib) [ 1712.977456] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1713.316597] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1715.039323] Key type lgssc registered [ 1715.957910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.634186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1724.696328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.664224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.253357] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1736.164774] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.228425] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1741.291262] LustreError: 69475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.737569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.866567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1749.454422] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1759.059517] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1764.884052] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1771.942094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1772.824653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.786631] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1787.207558] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1805.936988] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1820.137648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.149543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.168369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.193036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.194730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.214524] Lustre: Skipped 1 previous similar message [ 1822.234506] Lustre: Skipped 2 previous similar messages [ 1824.693337] Lustre: server umount lustre-MDT0000 complete [ 1827.299368] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.330103] LustreError: 69474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1828.984817] LustreError: 69455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121218 with bad export cookie 10116904042105686234 [ 1828.987667] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.997375] LustreError: 69455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1829.441517] Lustre: server umount lustre-MDT0001 complete [ 1846.057594] Lustre: server umount lustre-OST0000 complete [ 1877.040254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.335822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1877.362252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.734627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.681224] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1892.105386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.212323] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1892.242821] Lustre: 74371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1892.962557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1898.445366] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1905.935961] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1912.741154] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1920.090781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1920.287871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1920.890515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.716359] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1937.268120] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1939.052612] Lustre: 76948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1940.858900] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1946.595073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.597926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.616333] LustreError: Skipped 1 previous similar message [ 1946.621682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.641215] Lustre: Skipped 1 previous similar message [ 1946.663598] Lustre: Skipped 1 previous similar message [ 1951.720399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.734203] Lustre: Skipped 1 previous similar message [ 1952.733993] Lustre: server umount lustre-OST0000 complete [ 1959.394618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1959.415150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.432189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.568517] Lustre: server umount lustre-MDT0000 complete [ 1967.077744] LustreError: 74382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.092036] LustreError: 74382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1967.211901] LustreError: 76238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121357 with bad export cookie 10116904042105687074 [ 1967.226181] LustreError: 76238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1967.226533] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.602263] Lustre: server umount lustre-MDT0001 complete [ 1980.347669] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1983.642468] Key type lgssc unregistered [ 1984.043623] LNet: 78163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.049513] LNetError: 78163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.069910] LNet: Removed LNI 192.168.202.107@tcp [ 1985.216482] Key type .llcrypt unregistered [ 1985.217895] Key type ._llcrypt unregistered [ 2013.111632] Key type ._llcrypt registered [ 2013.116785] Key type .llcrypt registered [ 2013.218435] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2014.371640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.503282] alg: No test for adler32 (adler32-zlib) [ 2015.928213] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2016.271891] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2018.015514] Key type lgssc registered [ 2019.531857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2027.874136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2027.894123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.628940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.927450] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2039.777854] LustreError: 79575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.793077] LustreError: 79575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2044.282917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.000968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2050.477478] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2058.299642] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.684937] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.870876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.536573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.642741] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2088.091824] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2094.028143] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2095.102584] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2096.896947] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2096.904615] Lustre: Skipped 1 previous similar message [ 2099.370689] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2099.375500] Lustre: Skipped 2 previous similar messages [ 2104.085935] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2104.094743] Lustre: Skipped 4 previous similar messages [ 2112.210318] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2112.227328] Lustre: Skipped 7 previous similar messages [ 2128.415665] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2128.423075] Lustre: Skipped 13 previous similar messages [ 2161.050930] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2161.053447] Lustre: Skipped 29 previous similar messages [ 2206.846464] Lustre: 87687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2210.443816] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:39:57 (1787121597) [ 2212.139965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2214.380714] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:40:01 (1787121601) [ 2223.075059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.078082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.095296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.107220] Lustre: Skipped 1 previous similar message [ 2223.120190] Lustre: Skipped 1 previous similar message [ 2228.195943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.206920] Lustre: Skipped 1 previous similar message [ 2233.317948] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.321393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2233.347403] Lustre: Skipped 1 previous similar message [ 2233.656906] Lustre: server umount lustre-OST0000 complete [ 2238.433466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.441650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.449655] Lustre: Skipped 1 previous similar message [ 2243.552588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.564330] Lustre: Skipped 1 previous similar message [ 2252.255910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2252.446686] Lustre: server umount lustre-MDT0000 complete [ 2253.795197] LustreError: 79570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.814457] LustreError: 79570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2256.690864] LustreError: 79556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121646 with bad export cookie 5102333897829250308 [ 2256.693734] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.708187] LustreError: 79556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2257.103404] Lustre: server umount lustre-MDT0001 complete [ 2268.658812] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2271.705800] Key type lgssc unregistered [ 2271.977568] LNet: 88942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.996362] LNetError: 88942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.014654] LNet: Removed LNI 192.168.202.107@tcp [ 2272.968570] Key type .llcrypt unregistered [ 2272.975757] Key type ._llcrypt unregistered [ 2278.449956] Key type ._llcrypt registered [ 2278.452158] Key type .llcrypt registered [ 2278.561948] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2283.988272] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules [ 2285.755424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.892542] alg: No test for adler32 (adler32-zlib) [ 2287.069308] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2287.454212] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2289.192722] Key type lgssc registered [ 2290.701971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2292.155962] LustreError: 90193:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2311.984342] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2314.635407] LustreError: 90193:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2323.240070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.271713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.845570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.204901] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2334.185867] LustreError: 91217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.206223] LustreError: 91217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2339.297410] LustreError: 91218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.139901] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:42:07 (1787121727) [ 2344.420548] LustreError: 91217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.202368] LustreError: 91249:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2345.219255] LustreError: 91249:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2345.227648] LustreError: 91249:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2345.526212] Lustre: server umount lustre-MDT0000 complete [ 2361.043515] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2364.536566] Key type lgssc unregistered [ 2364.932877] LNet: 92746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.943495] LNetError: 92746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.963224] LNet: Removed LNI 192.168.202.107@tcp [ 2366.098417] Key type .llcrypt unregistered [ 2366.101828] Key type ._llcrypt unregistered [ 2388.618901] Key type ._llcrypt registered [ 2388.624995] Key type .llcrypt registered [ 2388.808812] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2391.052390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2391.123036] alg: No test for adler32 (adler32-zlib) [ 2392.525756] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2392.982791] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2394.671240] Key type lgssc registered [ 2396.443463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2406.297308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.313263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.020031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.096481] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2417.124495] LustreError: 93968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.143574] LustreError: 93968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2422.256107] LustreError: 93967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.846872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.840504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.941529] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2439.941697] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2445.539666] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2452.898101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.734727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.444089] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2467.343875] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2533.190627] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:45:20 (1787121920) [ 2535.357637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2537.516730] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:45:25 (1787121925) [ 2543.079959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.095494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.115308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.099119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.099768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.120744] Lustre: Skipped 1 previous similar message [ 2544.143637] Lustre: Skipped 2 previous similar messages [ 2548.136905] Lustre: server umount lustre-MDT0000 complete [ 2549.223620] LustreError: 93964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.253627] LustreError: 93964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2553.073984] LustreError: 93948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121943 with bad export cookie 1521931401282151315 [ 2553.087444] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.579900] Lustre: server umount lustre-MDT0001 complete [ 2569.080828] Lustre: server umount lustre-OST0000 complete [ 2581.892078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.531311] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.779851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.138996] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.464851] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2597.344908] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.359892] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2599.126392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.050854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.956232] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2613.074084] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2619.098867] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2633.705507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.890785] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2633.897579] LustreError: 98276:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2633.922860] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2633.932765] LustreError: 100371:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2633.939810] LustreError: 100371:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2633.944461] LustreError: 100371:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2633.997766] Lustre: server umount lustre-OST0000 complete [ 2634.000515] LustreError: 100371:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2643.025306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.195180] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2643.208606] LustreError: 98277:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2643.232740] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2643.240655] LustreError: 100838:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2643.253156] LustreError: 100838:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2643.262716] LustreError: 100838:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2643.373774] Lustre: server umount lustre-OST0000 complete [ 2643.382292] LustreError: 100838:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2654.247413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.452267] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2654.459667] Lustre: Found index 0 for lustre-OST0000, updating log [ 2654.464836] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2654.527516] Lustre: lustre-OST0000: new disk, initializing [ 2654.531300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.540903] Lustre: 101372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2654.824873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.726583] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2665.728862] Lustre: Failing over lustre-OST0000 [ 2665.932583] Lustre: server umount lustre-OST0000 complete [ 2670.051904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.056809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.059907] Lustre: Skipped 1 previous similar message [ 2676.050963] Lustre: server umount lustre-MDT0000 complete [ 2676.195143] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.214130] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2680.570149] LustreError: 100391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122070 with bad export cookie 1521931401282154066 [ 2680.598108] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.605288] LustreError: 100391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2681.216636] Lustre: server umount lustre-MDT0001 complete [ 2691.034431] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:47:58 (1787122078) [ 2710.808910] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2722.304583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.950557] LustreError: 103735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.151395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.690047] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2737.974306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.731727] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2747.735169] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2756.328870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.401701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.416841] Lustre: Skipped 1 previous similar message [ 2766.342170] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2775.526735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.612707] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2775.706102] Lustre: lustre-OST0001: new disk, initializing [ 2775.712694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2775.720079] Lustre: 106053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2782.815212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2782.826738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2782.994725] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2782.995435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2790.649243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2800.403396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2805.225928] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2805.235470] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.254567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2805.276688] Lustre: Skipped 3 previous similar messages [ 2811.258685] LustreError: 103732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.277727] LustreError: 103732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2811.520402] Lustre: server umount lustre-MDT0001 complete [ 2820.854863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.952397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2820.958401] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2820.982886] Lustre: 103726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2821.476213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2821.479068] Lustre: Skipped 1 previous similar message [ 2822.246706] LustreError: 107607:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2822.257508] LustreError: 107607:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2822.264595] LustreError: 103728:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2822.389210] LustreError: 107628:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.107@tcp: cfg command failed: rc = -17 [ 2822.397819] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2822.397819] [ 2825.846460] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2826.751809] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2837.717945] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:50:24 (1787122224) [ 2859.107739] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2880.885902] Lustre: 109669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2903.278791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2907.298332] Lustre: 110804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2984.930357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2984.939274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.966254] Lustre: Skipped 2 previous similar messages [ 2984.973916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.978504] Lustre: Skipped 4 previous similar messages [ 2991.170948] Lustre: server umount lustre-MDT0000 complete [ 2995.456426] LustreError: 106772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122385 with bad export cookie 1521931401282155053 [ 2995.460782] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2995.473935] LustreError: 106772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2995.685648] LustreError: 103731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.686286] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.692537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2995.692547] Lustre: Skipped 8 previous similar messages [ 2995.702756] LustreError: 103731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2995.750440] Lustre: Skipped 4 previous similar messages [ 2995.974727] Lustre: server umount lustre-MDT0001 complete [ 3002.473082] Lustre: server umount lustre-OST0000 complete [ 3006.804674] Lustre: server umount lustre-OST0001 complete [ 3013.282761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3022.543772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3031.191776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3041.364038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3062.526710] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3072.211654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.234251] Lustre: lustre-MDT0000: reset Object Index mappings [ 3072.847803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.845134] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3084.949069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.970098] Lustre: lustre-MDT0001: reset Object Index mappings [ 3089.864733] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3092.886436] Lustre: 115896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3100.161981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.162763] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3111.089417] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3115.445477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.088403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3118.102683] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3118.230950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3121.558973] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3128.532939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3132.476383] Lustre: 117772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3178.843787] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:56:06 (1787122566) [ 3208.163967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3208.194586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.211010] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.219940] Lustre: Skipped 1 previous similar message [ 3211.615321] Lustre: server umount lustre-MDT0000 complete [ 3216.900022] LustreError: 117084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122606 with bad export cookie 1521931401282201911 [ 3216.908108] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.930582] LustreError: 117084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3224.083700] Lustre: server umount lustre-MDT0001 complete [ 3229.475398] LustreError: 118531:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x13a:0x0].0x0 (ffffa05348bfe700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3235.751835] Lustre: server umount lustre-OST0000 complete [ 3246.131553] Lustre: server umount lustre-OST0001 complete [ 3253.359435] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 3263.313760] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3300.881974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.186279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3301.216447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3301.298122] Lustre: lustre-MDT0000: new disk, initializing [ 3301.437508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3306.464298] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3313.379875] Lustre: server umount lustre-MDT0000 complete [ 3325.478549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.953513] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3325.986507] Lustre: 122042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3326.015798] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3332.978277] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3339.266488] Lustre: server umount lustre-MDT0000 complete [ 3350.726729] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:58:57 (1787122737) [ 3352.944456] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3354.987694] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:59:02 (1787122742) [ 3374.956937] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3385.855710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.352449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3386.364187] Lustre: Skipped 5 previous similar messages [ 3391.551994] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3401.094155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.263871] Lustre: 124094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3401.377240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3401.391241] Lustre: Skipped 1 previous similar message [ 3401.519316] Lustre: lustre-MDT0001: new disk, initializing [ 3401.753401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3401.758600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3406.472304] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3412.413133] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3420.041509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.386684] Lustre: lustre-OST0000: new disk, initializing [ 3420.389771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3420.405137] Lustre: 125671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3421.664504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3421.679380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3421.833865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3428.467254] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3440.534803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.731718] Lustre: lustre-OST0001: new disk, initializing [ 3440.733581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3440.745189] Lustre: 126542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3442.648802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3442.659619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3442.871636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3447.949191] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3460.418623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3465.376501] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3881.170852] Lustre: 132973:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3881.190880] Lustre: 132973:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3881.199252] Lustre: 132973:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3881.210995] Lustre: 132973:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3881.221621] Lustre: 132973:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3881.230485] Lustre: 132973:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4177.113448] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:12:43 (1787123563) [ 4195.317548] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.342155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4195.346138] Lustre: Skipped 7 previous similar messages [ 4195.346428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.346431] Lustre: Skipped 8 previous similar messages [ 4197.326941] Lustre: server umount lustre-MDT0000 complete [ 4200.431981] LustreError: 127358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.473228] LustreError: 127358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4202.023206] LustreError: 133007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123591 with bad export cookie 1521931401282268019 [ 4202.037347] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.041910] LustreError: 133007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4205.537741] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.545333] Lustre: Skipped 2 previous similar messages [ 4205.547948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4205.571050] Lustre: Skipped 4 previous similar messages [ 4208.828356] Lustre: server umount lustre-MDT0001 complete [ 4219.292855] Lustre: server umount lustre-OST0000 complete [ 4229.520153] Lustre: server umount lustre-OST0001 complete [ 4237.899942] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 4249.893869] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4294.680896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.078604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4295.129395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4295.188336] Lustre: lustre-MDT0000: new disk, initializing [ 4295.259484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.263115] Lustre: Skipped 3 previous similar messages [ 4295.275705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4300.562882] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4312.781697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.111432] Lustre: 136033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4313.154713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4313.157855] Lustre: Skipped 1 previous similar message [ 4313.234558] Lustre: lustre-MDT0001: new disk, initializing [ 4313.383227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4313.400193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4318.804086] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4330.194401] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4336.753877] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.978957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.353860] Lustre: lustre-OST0000: new disk, initializing [ 4344.357397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4344.362624] Lustre: 138000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4345.962423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4345.976285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4346.178397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4352.440237] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4363.513246] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4376.115365] Lustre: lustre-MDD0000: changelog on [ 4379.334965] Lustre: lustre-MDD0001: changelog on [ 6208.058705] Lustre: lustre-MDD0001: changelog off [ 6210.151277] Lustre: 136039:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6211.485682] Lustre: lustre-MDD0000: changelog off [ 6220.257183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.260445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.265837] LustreError: Skipped 1 previous similar message [ 6220.272110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6220.272119] Lustre: Skipped 2 previous similar messages [ 6220.306792] Lustre: Skipped 1 previous similar message [ 6225.394430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6225.405491] Lustre: Skipped 2 previous similar messages [ 6225.899560] Lustre: server umount lustre-OST0000 complete [ 6230.497449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6230.513499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6230.525252] Lustre: Skipped 1 previous similar message [ 6235.617251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6235.626722] Lustre: Skipped 1 previous similar message [ 6236.898634] Lustre: server umount lustre-MDT0000 complete [ 6240.742017] LustreError: 136893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.772332] LustreError: 136893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6241.818634] LustreError: 136024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125631 with bad export cookie 1521931401283085766 [ 6241.832115] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6241.835394] LustreError: 136024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6242.569218] Lustre: server umount lustre-MDT0001 complete [ 6259.952577] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 6265.273991] Key type lgssc unregistered [ 6265.703118] LNet: 147138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6265.715748] LNetError: 147138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6266.757637] LNet: Removed LNI 192.168.202.107@tcp [ 6268.364180] Key type .llcrypt unregistered [ 6268.370120] Key type ._llcrypt unregistered [ 6294.526271] Key type ._llcrypt registered [ 6294.529492] Key type .llcrypt registered [ 6294.632373] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 6318.035234] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6319.566985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6319.712930] alg: No test for adler32 (adler32-zlib) [ 6321.285432] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 6321.664775] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 6323.447181] Key type lgssc registered [ 6325.266069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6368.605535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6368.646212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.896560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6369.932407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6370.022285] Lustre: lustre-MDT0000: new disk, initializing [ 6370.128929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.139988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6375.753746] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6390.456396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.663834] Lustre: 150345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6390.715602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6390.722470] Lustre: Skipped 1 previous similar message [ 6390.807112] Lustre: lustre-MDT0001: new disk, initializing [ 6390.964881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6390.997823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6391.007070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6395.669430] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6406.534442] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6411.651292] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6421.660905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.287099] Lustre: lustre-OST0000: new disk, initializing [ 6422.306091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6422.318756] Lustre: 152315:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6422.495753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.329755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6431.368172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6431.626189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6431.732229] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6443.478156] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6450.055801] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_module obdecho/obdecho [ 6452.331273] LustreError: 153317:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6452.351495] LustreError: 153317:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6454.458728] LustreError: 153414:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6472.682744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.695936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6475.533503] Lustre: server umount lustre-MDT0000 complete [ 6480.327312] LustreError: 150337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125870 with bad export cookie 16905635313602691549 [ 6480.330856] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6480.335332] LustreError: 150337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6480.718544] Lustre: server umount lustre-MDT0001 complete [ 6496.526245] Lustre: server umount lustre-OST0000 complete [ 6522.556540] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 6537.354606] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6583.829543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.130921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6584.187242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6584.316731] Lustre: lustre-MDT0000: new disk, initializing [ 6584.416150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.439736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6589.912096] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6601.910461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.073976] Lustre: 157133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.128509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6602.131708] Lustre: Skipped 1 previous similar message [ 6602.196847] Lustre: lustre-MDT0001: new disk, initializing [ 6602.283573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6602.314414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6602.326975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6607.985257] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6619.650368] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.732876] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6635.913532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.445422] Lustre: lustre-OST0000: new disk, initializing [ 6636.460835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6636.468879] Lustre: 159103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6636.748402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.731099] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6646.486482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6646.510894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6646.663562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6654.600534] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.299229] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6660.431089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6667.311646] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6667.647355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6672.353410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6672.360495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.370752] LustreError: Skipped 1 previous similar message [ 6672.375602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.375611] Lustre: Skipped 2 previous similar messages [ 6672.431323] Lustre: Skipped 3 previous similar messages [ 6675.816680] Lustre: server umount lustre-OST0000 complete [ 6682.606374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.610669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.625157] Lustre: Skipped 1 previous similar message [ 6682.637112] Lustre: Skipped 2 previous similar messages [ 6686.662209] Lustre: server umount lustre-MDT0000 complete [ 6687.715556] LustreError: 157146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.738654] LustreError: 157146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6690.908326] LustreError: 157127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126080 with bad export cookie 16905635313602692508 [ 6690.915165] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.916418] LustreError: 157127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6691.280929] Lustre: server umount lustre-MDT0001 complete [ 6695.141303] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:54:42 (1787126082) [ 6697.143518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6699.572754] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:54:47 (1787126087) [ 6708.964622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.681328] LustreError: 161520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.837264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.861730] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.494610] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6719.972171] LustreError: 161520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.092993] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.220733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.445649] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6740.641745] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6748.101686] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6756.468013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.858604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.874274] Lustre: Skipped 1 previous similar message [ 6765.947928] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6775.944588] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.097022] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 6792.166054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6792.177152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.702662] Lustre: server umount lustre-MDT0000 complete [ 6794.751927] LustreError: 164013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.784355] LustreError: 164013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6799.232228] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126189 with bad export cookie 16905635313602693390 [ 6799.249061] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.842296] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.859683] Lustre: Skipped 2 previous similar messages [ 6799.875322] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6799.877917] Lustre: Skipped 1 previous similar message [ 6805.775430] Lustre: server umount lustre-MDT0001 complete [ 6814.066061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6830.932795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.368457] LustreError: 165385:0:(llog.c:1655:llog_backup()) MGC192.168.202.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6831.383537] Lustre: 165385:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6832.057815] LustreError: 165390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.079433] LustreError: 165390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6865.376478] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.418058] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 6892.001671] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6892.010913] Lustre: 165385:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6930.400418] LustreError: 165391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6930.416959] LustreError: 165391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 6933.014968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6938.116587] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6939.653175] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6950.608898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.614600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6970.850158] LustreError: 148565:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000f72eb25 x1873937129680128/t0(0) o250->MGC192.168.202.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6976.492685] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6978.108169] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6987.362980] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.825471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6997.422314] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.408981] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7022.521401] Lustre: server umount lustre-MDT0000 complete [ 7022.566159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.570475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7027.059479] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126416 with bad export cookie 16905635313602699879 [ 7027.068881] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7027.079393] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7027.722926] Lustre: server umount lustre-MDT0001 complete [ 7042.763180] Lustre: server umount lustre-OST0000 complete [ 7070.547800] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7085.750421] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7131.739553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.328904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7132.386127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7132.498327] Lustre: lustre-MDT0000: new disk, initializing [ 7132.758193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.767432] Lustre: Skipped 1 previous similar message [ 7132.805956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7140.247871] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7154.061901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.202986] Lustre: 170529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7154.613392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7160.581942] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7173.288347] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7180.094995] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7187.482461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.702401] Lustre: lustre-OST0000: new disk, initializing [ 7187.706801] Lustre: Skipped 1 previous similar message [ 7187.712287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7187.717464] Lustre: Skipped 2 previous similar messages [ 7187.722303] Lustre: 172500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7197.096275] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7197.347344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7197.368599] Lustre: Skipped 1 previous similar message [ 7197.404472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7197.790939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7211.549604] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7218.334563] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7218.810746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7223.982800] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7224.175362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7228.387265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.395775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7228.406964] Lustre: Skipped 3 previous similar messages [ 7228.418157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7228.431940] Lustre: Skipped 3 previous similar messages [ 7232.839143] Lustre: server umount lustre-OST0000 complete [ 7244.795192] Lustre: server umount lustre-MDT0000 complete [ 7248.876559] LustreError: 170539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.895184] LustreError: 170539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7249.188762] LustreError: 171324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126639 with bad export cookie 16905635313602716686 [ 7249.190989] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.203781] LustreError: 171324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7249.647963] Lustre: server umount lustre-MDT0001 complete [ 7259.752363] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 04:04:07 (1787126647) [ 7269.305754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.610754] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7290.511967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.230451] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7305.047541] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7311.976535] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7319.947915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7328.545359] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7337.441874] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7356.466515] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 04:05:43 (1787126743) [ 7362.529494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7362.541861] LustreError: Skipped 1 previous similar message [ 7362.543656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.564207] Lustre: Skipped 2 previous similar messages [ 7362.573063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7362.579381] Lustre: Skipped 4 previous similar messages [ 7367.154869] Lustre: server umount lustre-OST0000 complete [ 7377.594843] Lustre: server umount lustre-MDT0000 complete [ 7381.736234] LustreError: 174897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126771 with bad export cookie 16905635313602717568 [ 7381.750754] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7382.395220] Lustre: server umount lustre-MDT0001 complete [ 7396.326524] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7399.883395] Key type lgssc unregistered [ 7400.269500] LNet: 178668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7400.274321] LNetError: 178668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7400.290344] LNet: Removed LNI 192.168.202.107@tcp [ 7401.494204] Key type .llcrypt unregistered [ 7401.496059] Key type ._llcrypt unregistered [ 7423.390594] Key type ._llcrypt registered [ 7423.393681] Key type .llcrypt registered [ 7423.519642] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7425.263695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7425.290782] alg: No test for adler32 (adler32-zlib) [ 7426.455426] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7426.790400] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7428.463427] Key type lgssc registered [ 7429.737988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7439.353719] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7439.377412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.944348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7460.063349] LustreError: 179915:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.107@tcp: failed processing log, type 4: rc = -110 [ 7465.247793] LustreError: 179915:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7465.269113] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7465.274340] LustreError: MGC192.168.202.107@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7465.287286] LustreError: 179915:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7465.295367] LustreError: 179915:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7465.319294] LustreError: 179915:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7465.391564] Lustre: server umount lustre-OST0000 complete [ 7465.394034] LustreError: 179915:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7475.899208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.578348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.830892] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7487.974916] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.000566] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7492.205595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.562371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7497.200723] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7506.137520] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7513.422420] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7521.249607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.406433] LustreError: 182232:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7521.418317] Lustre: 182232:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.107@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7521.549204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.743263] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7539.080026] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.266166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.271218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.295606] Lustre: Skipped 1 previous similar message [ 7543.306308] Lustre: Skipped 2 previous similar messages [ 7548.310746] Lustre: server umount lustre-MDT0000 complete [ 7548.386581] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.420842] LustreError: 180415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7552.601688] LustreError: 180395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126942 with bad export cookie 8105835134277008462 [ 7552.610845] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7552.863521] Lustre: server umount lustre-MDT0001 complete [ 7569.192621] Lustre: server umount lustre-OST0000 complete [ 7581.205338] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7593.786678] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7635.110566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7635.402725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7635.432512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7635.500707] Lustre: lustre-MDT0000: new disk, initializing [ 7635.564891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.577763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7641.039599] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7652.164578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.333573] Lustre: 185760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7652.383421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7652.386511] Lustre: Skipped 1 previous similar message [ 7652.478251] Lustre: lustre-MDT0001: new disk, initializing [ 7652.607059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.655255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7652.671575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7658.229992] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7667.837126] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7674.059761] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7681.600708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.975830] Lustre: lustre-OST0000: new disk, initializing [ 7681.988141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7682.001101] Lustre: 187730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7682.133037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.987414] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7690.411371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7690.434992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7690.601256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7701.434505] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7707.534443] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7707.791765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7712.886486] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7713.104388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7716.330063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7716.340875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.344926] LustreError: Skipped 1 previous similar message [ 7716.358608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7716.383733] Lustre: Skipped 2 previous similar messages [ 7716.407762] Lustre: Skipped 1 previous similar message [ 7721.454432] LustreError: 187715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.473372] LustreError: 187715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7721.647660] Lustre: server umount lustre-OST0000 complete [ 7726.561693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.582209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7726.596217] Lustre: Skipped 1 previous similar message [ 7731.683507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.695951] Lustre: Skipped 1 previous similar message [ 7732.124465] Lustre: server umount lustre-MDT0000 complete [ 7736.044622] LustreError: 185753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127125 with bad export cookie 8105835134277009183 [ 7736.055386] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7736.061273] LustreError: 185753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7736.428644] Lustre: server umount lustre-MDT0001 complete [ 7746.246152] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 04:12:13 (1787127133) [ 7765.217389] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7768.749455] Key type lgssc unregistered [ 7769.143161] LNet: 190682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7769.154953] LNetError: 190682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7769.185137] LNet: Removed LNI 192.168.202.107@tcp [ 7770.280234] Key type .llcrypt unregistered [ 7770.282340] Key type ._llcrypt unregistered [ 7790.956321] Key type ._llcrypt registered [ 7790.957928] Key type .llcrypt registered [ 7791.206903] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7793.041672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7793.054592] alg: No test for adler32 (adler32-zlib) [ 7794.143715] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7794.471299] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7796.191207] Key type lgssc registered [ 7797.851190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7807.634631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7807.653355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.221882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.154615] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7819.757638] LustreError: 191899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.796739] LustreError: 191899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7823.353408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7823.736818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7830.072045] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7838.016334] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7843.218912] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7852.050565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.202038] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7852.206217] LustreError: 193764:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7852.214150] Lustre: 193764:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.107@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7852.335235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.353814] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7872.251271] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7875.041327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.055537] Lustre: Skipped 1 previous similar message [ 7875.056405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.064410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7875.093498] Lustre: Skipped 2 previous similar messages [ 7880.181929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.190700] Lustre: Skipped 2 previous similar messages [ 7881.033744] Lustre: server umount lustre-MDT0000 complete [ 7884.532747] LustreError: 191882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127274 with bad export cookie 15834754708608404464 [ 7884.539343] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7884.544198] LustreError: 191882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7884.781546] Lustre: server umount lustre-MDT0001 complete [ 7898.233510] Lustre: server umount lustre-OST0000 complete [ 7907.935898] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7920.278186] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7956.652806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.895994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7956.918353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7956.977788] Lustre: lustre-MDT0000: new disk, initializing [ 7957.029340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.041717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7961.381509] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7972.997046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.180323] Lustre: 197290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7973.226694] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7973.228703] Lustre: Skipped 1 previous similar message [ 7973.233588] LustreError: 198073:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7973.243955] LustreError: 198073:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7973.250291] Lustre: 198073:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7973.274268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7973.278612] Lustre: Skipped 1 previous similar message [ 7973.401995] Lustre: lustre-MDT0001: new disk, initializing [ 7973.584752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7973.656073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7973.671967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7978.251259] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7987.660901] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7993.273359] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8000.928318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.170968] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8001.172525] Lustre: Skipped 1 previous similar message [ 8001.182899] LustreError: 199234:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8001.189120] LustreError: 199234:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8001.194681] Lustre: 199234:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8001.200382] Lustre: 199234:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8001.217348] Lustre: lustre-OST0000: new disk, initializing [ 8001.220427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8001.225221] Lustre: 199256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8001.317655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8007.820707] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8010.881204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8010.897096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8010.957549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8018.691107] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.742288] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8023.967913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8028.774464] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8029.014606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8031.714108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8031.716673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.727090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.751926] Lustre: Skipped 2 previous similar messages [ 8036.837250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8036.859224] Lustre: Skipped 2 previous similar messages [ 8038.034353] Lustre: server umount lustre-OST0000 complete [ 8045.026087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8045.032672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.041143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.087038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.342380] Lustre: server umount lustre-MDT0000 complete [ 8052.196200] LustreError: 197297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.223206] LustreError: 197297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8052.837773] LustreError: 197284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127442 with bad export cookie 15834754708608405185 [ 8052.849422] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.883145] LustreError: 197284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8053.610315] Lustre: server umount lustre-MDT0001 complete [ 8063.523854] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 04:17:31 (1787127451) [ 8083.200895] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8094.623603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8095.055418] LustreError: 202386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.195538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8100.320931] LustreError: 202387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.866078] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8105.440941] LustreError: 202386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.932149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.989249] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8125.963681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8132.970749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8133.228809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.235563] Lustre: Skipped 1 previous similar message [ 8140.585079] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8149.204986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.386939] Lustre: lustre-OST0001: new disk, initializing [ 8149.391316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8149.398279] Lustre: 204696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8156.519459] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8156.847245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8156.858572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8156.916887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8167.198305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8176.776620] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8178.793899] Lustre: DEBUG MARKER: ADD OST3 [ 8184.294614] loop: module loaded [ 8184.379100] loop0: detected capacity change from 0 to 204800000 [ 8184.626456] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8184.649177] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8184.678093] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8191.684334] loop0: detected capacity change from 0 to 204800000 [ 8192.155444] Lustre: DEBUG MARKER: STOP OST3 [ 8199.146571] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.338446] Lustre: lustre-OST0003: new disk, initializing [ 8199.341326] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8199.346504] Lustre: 206744:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8199.475075] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8199.478083] Lustre: Skipped 1 previous similar message [ 8201.130748] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8201.383620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8201.396334] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8201.462518] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8202.980778] Lustre: Failing over lustre-OST0003 [ 8203.176468] LustreError: 206876:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8203.189309] Lustre: server umount lustre-OST0003 complete [ 8206.320544] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8206.325796] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.330130] LustreError: 204227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8206.330143] LustreError: 204227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8206.379960] Lustre: Skipped 1 previous similar message [ 8207.278743] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8216.442600] Lustre: DEBUG MARKER: START OST3 again [ 8223.421020] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8223.816244] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8225.644103] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8225.900863] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8225.904367] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8229.737824] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8249.960165] Lustre: Failing over lustre-OST0003 [ 8250.165552] Lustre: server umount lustre-OST0003 complete [ 8251.874363] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8251.874751] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.888273] LustreError: Skipped 1 previous similar message [ 8251.892391] LustreError: 204227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.903927] Lustre: Skipped 1 previous similar message [ 8251.938247] LustreError: 204227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8269.280978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8269.288469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.304649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8269.312164] Lustre: Skipped 2 previous similar messages [ 8274.701944] Lustre: server umount lustre-MDT0000 complete [ 8278.202713] LustreError: 205462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127668 with bad export cookie 15834754708608406067 [ 8278.204424] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8278.215855] LustreError: 205462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8284.700328] Lustre: server umount lustre-MDT0001 complete [ 8288.926891] Lustre: server umount lustre-OST0000 complete [ 8293.056290] Lustre: server umount lustre-OST0001 complete [ 8300.257824] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8309.695812] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8347.598370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.797993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8347.825539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8347.902722] Lustre: lustre-MDT0000: new disk, initializing [ 8347.979642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.984772] Lustre: Skipped 1 previous similar message [ 8347.991887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8352.736219] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8363.292880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.447911] Lustre: 212158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8363.722739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8367.929978] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8376.696774] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.817438] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8388.240185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.494541] Lustre: lustre-OST0000: new disk, initializing [ 8388.497816] Lustre: Skipped 1 previous similar message [ 8388.508487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8388.516274] Lustre: Skipped 2 previous similar messages [ 8388.520702] Lustre: 214125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8389.998621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8390.007937] Lustre: Skipped 1 previous similar message [ 8390.015245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8390.154953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8395.264933] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8404.488228] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8408.944038] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8409.189350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8415.577143] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8415.860855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8420.845217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8420.855243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.865296] Lustre: Skipped 5 previous similar messages [ 8420.871298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8420.896269] Lustre: Skipped 8 previous similar messages [ 8424.173763] Lustre: server umount lustre-OST0000 complete [ 8434.639118] Lustre: server umount lustre-MDT0000 complete [ 8436.209623] LustreError: 212170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.238071] LustreError: 212170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8437.987900] LustreError: 212151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127827 with bad export cookie 15834754708608408363 [ 8437.996961] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.004462] LustreError: 212151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8438.673996] Lustre: server umount lustre-MDT0001 complete [ 8443.073964] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 04:23:50 (1787127830) [ 8460.737828] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8472.706487] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8513.511825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.783498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8513.810817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8513.904424] Lustre: lustre-MDT0000: new disk, initializing [ 8514.033569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8519.349377] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8530.894800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.070037] Lustre: 218736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8531.416317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8536.596268] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8545.747720] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8551.196922] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8558.438409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.785487] Lustre: 220702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8560.354851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8565.985981] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8575.589487] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.341817] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8580.483194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8584.998520] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8585.204295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8591.332373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.335572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.353317] Lustre: Skipped 4 previous similar messages [ 8591.359102] Lustre: Skipped 3 previous similar messages [ 8601.055199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8601.424311] Lustre: server umount lustre-OST0000 complete [ 8611.815943] LustreError: 218747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8611.825230] LustreError: 218747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8611.875589] Lustre: server umount lustre-MDT0000 complete [ 8615.627807] LustreError: 218728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128005 with bad export cookie 15834754708608409245 [ 8615.637122] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8615.639651] LustreError: 218728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8616.165553] Lustre: server umount lustre-MDT0001 complete [ 8625.358344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8626.026410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.032107] Lustre: Skipped 5 previous similar messages [ 8631.028617] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8641.375057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.776744] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8652.905718] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8659.042716] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8666.207883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.287688] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8680.199090] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8691.169592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.180760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.187035] LustreError: Skipped 2 previous similar messages [ 8691.196151] Lustre: Skipped 5 previous similar messages [ 8696.233538] Lustre: server umount lustre-OST0000 complete [ 8714.207147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8714.499456] Lustre: server umount lustre-MDT0000 complete [ 8718.030034] LustreError: 222904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128107 with bad export cookie 15834754708608410120 [ 8718.049979] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.398401] Lustre: server umount lustre-MDT0001 complete [ 8821.198512] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8823.987166] Key type lgssc unregistered [ 8824.394997] LNet: 226591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8824.414489] LNetError: 226591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8824.433737] LNet: Removed LNI 192.168.202.107@tcp [ 8826.180167] Key type .llcrypt unregistered [ 8826.183204] Key type ._llcrypt unregistered [ 8858.417750] Key type ._llcrypt registered [ 8858.419992] Key type .llcrypt registered [ 8858.520380] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8874.746922] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8875.968272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8875.998464] alg: No test for adler32 (adler32-zlib) [ 8877.628585] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8878.207960] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8879.999492] Key type lgssc registered [ 8881.336360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8921.105843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8921.144915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.464906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.507527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8922.597070] Lustre: lustre-MDT0000: new disk, initializing [ 8922.712568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.743696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8927.694850] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8940.099470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.251359] Lustre: 229963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8940.349656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8940.359283] Lustre: Skipped 1 previous similar message [ 8940.525469] Lustre: lustre-MDT0001: new disk, initializing [ 8940.707431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8940.779906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8940.821631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8946.957600] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8959.360729] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8966.515552] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8974.249255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.754945] Lustre: lustre-OST0000: new disk, initializing [ 8974.758646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8974.764733] Lustre: 231931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8974.923859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.635339] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8984.645920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8984.653913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8984.805701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8994.081465] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9000.018071] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9000.297877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9005.196384] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9005.470236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9010.659776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9010.672457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.680276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9010.692356] Lustre: Skipped 1 previous similar message [ 9010.721923] Lustre: Skipped 1 previous similar message [ 9013.976698] Lustre: server umount lustre-OST0000 complete [ 9017.824387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9017.836371] LustreError: Skipped 1 previous similar message [ 9017.849817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.878325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9020.899516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.907966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9020.934302] Lustre: Skipped 1 previous similar message [ 9026.020225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.041308] Lustre: Skipped 1 previous similar message [ 9030.114788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.122287] Lustre: Skipped 1 previous similar message [ 9032.160801] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9032.391899] Lustre: server umount lustre-MDT0000 complete [ 9036.447918] LustreError: 229954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128426 with bad export cookie 15684859841296991436 [ 9036.452149] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9036.461838] LustreError: 229954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9036.995992] Lustre: server umount lustre-MDT0001 complete [ 9058.302160] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9070.384544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.866330] LustreError: 234869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9070.886764] LustreError: 234869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9070.932249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.199117] LustreError: 234870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.690219] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9080.293376] LustreError: 234869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.408649] LustreError: 234870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9087.374818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9087.864184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9092.725933] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9096.356750] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9104.035817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9104.391779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.036975] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9122.426670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9122.768796] Lustre: lustre-OST0001: new disk, initializing [ 9122.781707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9122.800200] Lustre: 237176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9122.926187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9130.778711] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9133.218981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9133.229939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9133.390985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9144.055795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9153.271495] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9168.028352] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:35:55 (1787128555) [ 9174.506486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.512596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.526060] Lustre: Skipped 2 previous similar messages [ 9174.543036] Lustre: Skipped 3 previous similar messages [ 9177.319875] Lustre: server umount lustre-MDT0000 complete [ 9179.619726] LustreError: 234869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.640912] LustreError: 234869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9182.240572] LustreError: 234850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128572 with bad export cookie 15684859841296992318 [ 9182.250349] LustreError: 234850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9182.253266] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.683856] Lustre: server umount lustre-MDT0001 complete [ 9197.754447] Lustre: server umount lustre-OST0000 complete [ 9200.098513] Lustre: 228183:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787128574/real 1787128574] req@ffffa0536ffeb800 x1873939810864896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787128590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9200.132783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.146957] Lustre: Skipped 1 previous similar message [ 9202.569320] Lustre: server umount lustre-OST0001 complete [ 9210.696703] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9227.585856] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9273.102881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.451698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9273.481303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9273.539798] Lustre: lustre-MDT0000: new disk, initializing [ 9273.623336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.640840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9278.807962] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9290.176357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9290.362449] Lustre: 241148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9290.414427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9290.417437] Lustre: Skipped 1 previous similar message [ 9290.482041] Lustre: lustre-MDT0001: new disk, initializing [ 9290.629564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9290.645794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9295.666909] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9306.278970] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9313.003496] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9319.968357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.313505] Lustre: lustre-OST0000: new disk, initializing [ 9320.318790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9320.327964] Lustre: 243115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9322.034764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9322.064657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9322.166804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9327.723987] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9339.116496] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.792454] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9344.058797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9349.348572] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9349.602622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9353.186621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9353.194544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9353.201761] LustreError: Skipped 1 previous similar message [ 9353.207995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9353.241157] Lustre: Skipped 1 previous similar message [ 9357.990597] Lustre: server umount lustre-OST0000 complete [ 9368.951190] Lustre: server umount lustre-MDT0000 complete [ 9373.668640] LustreError: 241155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.694387] LustreError: 241155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9373.813437] LustreError: 241141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128763 with bad export cookie 15684859841296994453 [ 9373.814864] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.823473] LustreError: 241141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9374.328800] Lustre: server umount lustre-MDT0001 complete [ 9384.527373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.950955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.954503] Lustre: Skipped 2 previous similar messages [ 9389.010860] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9390.052616] LustreError: 245334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9390.066509] LustreError: 245334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9398.079870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.310730] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9412.482336] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9419.013374] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9426.045179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.693915] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9442.195904] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9592.290568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9592.299909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.305956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9592.315929] Lustre: Skipped 3 previous similar messages [ 9592.340678] Lustre: Skipped 6 previous similar messages [ 9594.910834] Lustre: server umount lustre-OST0000 complete [ 9606.030297] Lustre: server umount lustre-MDT0000 complete [ 9607.650853] LustreError: 246062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9607.678344] LustreError: 246062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9610.701928] LustreError: 245316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129000 with bad export cookie 15684859841296995335 [ 9610.711990] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.720040] LustreError: 245316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9611.369152] Lustre: server umount lustre-MDT0001 complete [ 9713.145616] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9716.971774] Key type lgssc unregistered [ 9717.601501] LNet: 248947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9717.622483] LNetError: 248947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9717.660702] LNet: Removed LNI 192.168.202.107@tcp [ 9719.124178] Key type .llcrypt unregistered [ 9719.126505] Key type ._llcrypt unregistered [ 9751.163723] Key type ._llcrypt registered [ 9751.166491] Key type .llcrypt registered [ 9751.355622] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9769.567253] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9770.829591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9770.937201] alg: No test for adler32 (adler32-zlib) [ 9772.248685] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9772.453274] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9774.143501] Key type lgssc registered [ 9775.780131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9834.599072] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9847.240968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9847.287467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9848.659991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9848.722230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9848.901279] Lustre: lustre-MDT0000: new disk, initializing [ 9849.057877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.090704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9854.031454] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9865.825338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.014960] Lustre: 253024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9866.097971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9866.103383] Lustre: Skipped 1 previous similar message [ 9866.177422] Lustre: lustre-MDT0001: new disk, initializing [ 9866.284627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9866.335876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9866.352397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9871.551796] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9876.978569] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9885.631867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.088284] Lustre: lustre-OST0000: new disk, initializing [ 9886.092885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9886.100027] Lustre: 254654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9886.241054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.496797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9893.509157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9893.627910] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9893.696165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9905.445630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.729335] Lustre: lustre-OST0001: new disk, initializing [ 9905.737561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9905.746260] Lustre: 255529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9905.895682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9911.436945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9911.457120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9911.578856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9913.384067] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9925.501089] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9934.405831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9947.618173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.622183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.630850] Lustre: Skipped 3 previous similar messages [ 9947.640812] Lustre: Skipped 3 previous similar messages [ 9952.755887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.005551] Lustre: server umount lustre-MDT0000 complete [ 9957.241228] LustreError: 253014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129347 with bad export cookie 5425688337677898787 [ 9957.249900] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9957.258887] LustreError: 253014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9957.713668] Lustre: server umount lustre-MDT0001 complete [ 9972.806702] Lustre: server umount lustre-OST0000 complete [ 9973.200056] Lustre: 250527:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129347/real 1787129347] req@ffffa0537631df80 x1873940748253312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9973.255883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9978.080885] Lustre: server umount lustre-OST0001 complete [ 9995.217736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10006.007230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10015.778524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10028.969915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10055.276783] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10068.347893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10068.371090] Lustre: lustre-MDT0000: reset Object Index mappings [10068.891115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10074.092023] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.116966] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10074.630026] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10078.195608] LustreError: 260541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.313475] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10085.267143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10085.311839] Lustre: lustre-MDT0001: reset Object Index mappings [10085.626065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10091.445566] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10095.881760] Lustre: 261683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10104.231299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10104.677453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10110.838546] LustreError: 262036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.002817] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10113.904567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10119.142258] LustreError: 262305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.163866] LustreError: 262305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10123.843245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.465333] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10133.775966] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10142.798689] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10147.382952] Lustre: 263554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10154.036450] Lustre: server umount lustre-MDT0000 complete [10154.981340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.986511] LustreError: 262058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.999512] Lustre: Skipped 2 previous similar messages [10155.036854] LustreError: 262058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10158.936286] LustreError: 260522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129548 with bad export cookie 5425688337677905577 [10158.968165] LustreError: 260522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10158.970148] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.499941] Lustre: server umount lustre-MDT0001 complete [10174.925747] Lustre: server umount lustre-OST0000 complete [10176.671993] Lustre: 250525:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129550/real 1787129550] req@ffffa0537736f100 x1873940748306560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10176.691032] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.706392] Lustre: Skipped 1 previous similar message [10179.645730] Lustre: server umount lustre-OST0001 complete [10204.182474] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:53:11 (1787129591) [10224.273841] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10237.943848] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10286.615392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.179843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10287.231693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10287.376710] Lustre: lustre-MDT0000: new disk, initializing [10287.531477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10287.538855] Lustre: Skipped 1 previous similar message [10287.576040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10294.032556] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10308.369889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.585173] Lustre: 267827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10308.678109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10308.685946] Lustre: Skipped 1 previous similar message [10308.806904] Lustre: lustre-MDT0001: new disk, initializing [10308.913975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10308.926880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10314.002374] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10325.071511] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10334.882963] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10345.054795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.472815] Lustre: lustre-OST0000: new disk, initializing [10345.479882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10345.491865] Lustre: 269800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10346.561588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10346.570556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10346.671599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10354.160299] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10365.548164] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10371.761457] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10372.153807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10378.350479] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10378.618534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10382.824766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.829139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10382.873738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10382.890235] Lustre: Skipped 4 previous similar messages [10387.324099] Lustre: server umount lustre-OST0000 complete [10393.066751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.082852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10393.083512] Lustre: Skipped 2 previous similar messages [10398.179481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.191307] Lustre: Skipped 2 previous similar messages [10406.367628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10406.718517] Lustre: server umount lustre-MDT0000 complete [10407.392376] LustreError: 268698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.420250] LustreError: 268698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10410.812608] LustreError: 267819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129800 with bad export cookie 5425688337677907670 [10410.814529] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10410.822603] LustreError: 267819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10411.323947] Lustre: server umount lustre-MDT0001 complete [10434.558195] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10447.452631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.857276] LustreError: 272740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10447.900495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10447.904528] Lustre: Skipped 2 previous similar messages [10453.623405] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10464.101639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.961351] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10476.542826] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10486.507205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10494.580823] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10505.355059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.564242] Lustre: lustre-OST0001: new disk, initializing [10505.566751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10505.571846] Lustre: 275049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10507.145721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10507.151190] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10507.227145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10514.135596] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10526.332547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10530.875540] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10535.994768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10536.359430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10542.033713] Lustre: server umount lustre-MDT0000 complete [10543.084162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.091390] LustreError: 272735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.102040] Lustre: Skipped 2 previous similar messages [10543.130598] LustreError: 272735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10546.405382] LustreError: 272721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129936 with bad export cookie 5425688337677908552 [10546.420664] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.429655] LustreError: 272721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10546.910592] Lustre: server umount lustre-MDT0001 complete [10561.540398] Lustre: server umount lustre-OST0000 complete [10564.267076] Lustre: 250527:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129938/real 1787129938] req@ffffa0534e81d500 x1873940748419456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10566.591938] Lustre: server umount lustre-OST0001 complete [10586.786375] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10598.055336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.812143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10598.823250] Lustre: Skipped 3 previous similar messages [10604.655776] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10614.927020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.011471] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10624.525970] Lustre: 279245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10631.942840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.638777] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10643.649534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10649.995882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.778619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10658.019437] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10667.011286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10672.505603] Lustre: 281135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10678.787568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10679.186160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10691.418157] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 05:01:19 (1787130079) [10707.424186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10707.431414] LustreError: Skipped 1 previous similar message [10707.440896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.457924] Lustre: Skipped 2 previous similar messages [10707.469784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.479335] Lustre: Skipped 2 previous similar messages [10713.939778] Lustre: server umount lustre-MDT0000 complete [10717.160781] LustreError: 278100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.173238] LustreError: 278100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [10719.371373] LustreError: 278084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130109 with bad export cookie 5425688337677914852 [10719.387256] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10719.392403] LustreError: 278084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10724.335738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10724.344671] Lustre: Skipped 7 previous similar messages [10725.913121] Lustre: server umount lustre-MDT0001 complete [10745.311365] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10745.562376] Lustre: server umount lustre-OST0000 complete [10749.645562] Lustre: server umount lustre-OST0001 complete [10758.896445] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10775.649495] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10830.732264] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10843.864564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.095541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10844.142579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10844.236248] Lustre: lustre-MDT0000: new disk, initializing [10844.390749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10850.449373] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10862.032121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.117105] Lustre: 285396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10862.425203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10862.429980] Lustre: Skipped 4 previous similar messages [10862.485114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10868.175887] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10880.960472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10881.433483] Lustre: 287034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10883.594274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10891.287338] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10905.872231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10906.050216] Lustre: 287906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10907.350661] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10907.359505] Lustre: Skipped 1 previous similar message [10907.471279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10914.247323] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10924.444929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10929.413760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10929.419064] Lustre: Skipped 1 previous similar message [10948.861229] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 05:05:36 (1787130336) [10953.716834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10953.728288] Lustre: Skipped 5 previous similar messages [10954.011440] Lustre: server umount lustre-MDT0000 complete [10958.423518] LustreError: 285390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130348 with bad export cookie 5425688337677922727 [10958.425367] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10958.444189] LustreError: 285390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10958.841463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10958.857344] Lustre: Skipped 1 previous similar message [10959.266787] Lustre: server umount lustre-MDT0001 complete [10965.327852] Lustre: server umount lustre-OST0000 complete [10971.084677] Lustre: server umount lustre-OST0001 complete [10982.546312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.154454] LustreError: 290149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.177382] LustreError: 290149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10988.807420] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10998.772478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11005.048867] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11013.192857] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11019.425702] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11027.156078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.983216] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11043.834274] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11063.813757] loop1: detected capacity change from 0 to 204800000 [11064.087038] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11064.109830] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11064.119522] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11069.607915] loop1: detected capacity change from 0 to 204800000 [11075.434563] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11075.541153] LustreError: 290138:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs19205: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11075.552262] LustreError: 290138:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs19205 [11075.556465] LustreError: 290138:0:(mgs_handler.c:517:mgs_target_reg()) fs19205-MDT0001: Error checking target index: rc = -13 [11075.561467] LustreError: fs19205-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11075.573158] LustreError: 293866:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11075.583259] LustreError: 293866:0:(tgt_mount.c:2129:server_put_super()) no obd fs19205-MDT0001 [11075.589987] LustreError: 293866:0:(tgt_mount.c:116:server_deregister_mount()) fs19205-MDT0001 not registered [11075.627832] Lustre: server umount fs19205-MDT0001 complete [11075.635256] LustreError: 293866:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11084.794499] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.117394] Lustre: Setting parameter fs19205-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19205-MDT0001 [11085.206278] Lustre: srv-fs19205-MDT0001: No data found on store. Initialize space. [11085.218347] Lustre: Skipped 4 previous similar messages [11085.311761] Lustre: fs19205-MDT0001: new disk, initializing [11085.317465] Lustre: Skipped 3 previous similar messages [11085.437632] Lustre: srv-fs19205-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11085.449844] Lustre: 294333:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19205-MDT0001-mdtlov: can't create new llog: rc = -115 [11085.462259] LustreError: 294333:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19205-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11091.111654] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11098.967428] Lustre: server umount fs19205-MDT0001 complete [11110.893105] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11113.949925] loop1: detected capacity change from 0 to 204800000 [11120.819070] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.100246] Lustre: 295901:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19205-MDT0001-mdtlov: can't create new llog: rc = -115 [11121.111557] LustreError: 295901:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19205-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11126.248431] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11126.471784] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11131.944802] Lustre: server umount fs19205-MDT0001 complete [11142.624323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11142.633857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.647198] Lustre: Skipped 5 previous similar messages [11142.654623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.123657] Lustre: server umount lustre-MDT0000 complete [11158.079828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.263682] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11163.648701] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11164.043749] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11173.538330] loop1: detected capacity change from 0 to 204800000 [11179.900633] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.459792] Lustre: 298350:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19205-MDT0001-mdtlov: can't create new llog: rc = -115 [11180.470553] LustreError: 298350:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19205-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11186.604717] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11192.918706] Lustre: server umount fs19205-MDT0001 complete [11218.401042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11232.223812] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11232.447810] Lustre: server umount lustre-OST0000 complete [11240.419836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11240.441083] LustreError: Skipped 1 previous similar message [11243.390095] Lustre: server umount lustre-MDT0000 complete [11247.951719] LustreError: 290131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130637 with bad export cookie 5425688337677925408 [11247.957273] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11247.963178] LustreError: 290131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11248.444637] Lustre: server umount lustre-MDT0001 complete [11260.794644] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11264.783401] Key type lgssc unregistered [11265.100332] LNet: 300701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11265.111987] LNetError: 300701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11266.151644] LNet: Removed LNI 192.168.202.107@tcp [11267.382205] Key type .llcrypt unregistered [11267.395466] Key type ._llcrypt unregistered [11290.409474] Key type ._llcrypt registered [11290.413693] Key type .llcrypt registered [11290.503399] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11291.575460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11291.771213] alg: No test for adler32 (adler32-zlib) [11292.809920] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11293.059181] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11294.791809] Key type lgssc registered [11296.216394] Lustre: Echo OBD driver; http://www.lustre.org/ [11304.434202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11304.528382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.339667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11311.970553] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11316.724958] LustreError: 301973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.775556] LustreError: 301973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11321.834401] LustreError: 301974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.256045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.887688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11327.145632] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11334.433823] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11341.365174] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11349.993804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11350.663442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11352.699291] LustreError: 303805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.187754] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.379453] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11363.310118] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.720605] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11373.540105] LustreError: 303910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.569230] LustreError: 303910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11386.171455] loop1: detected capacity change from 0 to 204800000 [11386.331636] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11386.339849] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11386.346741] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11390.341363] LustreError: 303805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.367508] LustreError: 303805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11391.191560] loop1: detected capacity change from 0 to 204800000 [11398.762379] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11398.975475] Lustre: fs20070-OST0000: new disk, initializing [11398.977351] Lustre: srv-fs20070-OST0000: No data found on store. Initialize space. [11398.980453] Lustre: 305710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11399.098069] Lustre: fs20070-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11405.716456] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11415.359887] Lustre: server umount fs20070-OST0000 complete [11424.753881] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.770991] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11429.047947] loop1: detected capacity change from 0 to 204800000 [11434.993479] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11435.221033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11435.729090] Lustre: fs20070-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11443.918868] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11450.352298] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11454.035406] loop2: detected capacity change from 0 to 204800000 [11454.333092] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11454.361526] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11454.375125] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.766337] loop2: detected capacity change from 0 to 204800000 [11466.830699] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11466.960511] LustreError: fs20070-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11466.970648] LustreError: fs20070-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11466.978841] LustreError: 308752:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11466.985364] LustreError: 308752:0:(tgt_mount.c:2129:server_put_super()) no obd fs20070-OST0001 [11466.999671] LustreError: 308752:0:(tgt_mount.c:116:server_deregister_mount()) fs20070-OST0001 not registered [11467.050664] Lustre: server umount fs20070-OST0001 complete [11467.052760] LustreError: 308752:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11478.104160] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.280843] Lustre: fs20070-OST0001: new disk, initializing [11478.283599] Lustre: srv-fs20070-OST0001: No data found on store. Initialize space. [11478.295996] Lustre: 309304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11478.411322] Lustre: fs20070-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11485.613536] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11491.321960] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.370410] LustreError: 303806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11495.633637] Lustre: server umount fs20070-OST0000 complete [11505.914530] Lustre: server umount fs20070-OST0001 complete [11521.776115] loop1: detected capacity change from 0 to 204800000 [11528.618119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11528.973709] Lustre: fs20070-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11536.419929] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11544.024637] loop2: detected capacity change from 0 to 204800000 [11550.923930] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11558.767695] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11570.174642] Lustre: server umount fs20070-OST0000 complete [11581.029979] Lustre: server umount fs20070-OST0001 complete [11588.596418] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11598.818201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11598.829946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11598.859604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11598.866757] Lustre: Skipped 1 previous similar message [11602.226148] Lustre: server umount lustre-OST0000 complete [11609.057199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11609.064205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11609.077943] Lustre: Skipped 2 previous similar messages [11609.090576] Lustre: Skipped 1 previous similar message [11613.196565] Lustre: server umount lustre-MDT0000 complete [11617.230733] LustreError: 301954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131007 with bad export cookie 6573446449560945507 [11617.240480] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11617.244029] LustreError: 301954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11618.132055] Lustre: server umount lustre-MDT0001 complete [11631.422292] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11634.230806] Key type lgssc unregistered [11634.555762] LNet: 315233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11634.575187] LNetError: 315233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.599539] LNet: Removed LNI 192.168.202.107@tcp [11635.426296] Key type .llcrypt unregistered [11635.429111] Key type ._llcrypt unregistered [11672.964354] Key type ._llcrypt registered [11672.967559] Key type .llcrypt registered [11673.184249] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [11690.173371] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11691.798769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11691.876978] alg: No test for adler32 (adler32-zlib) [11692.953970] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11693.183652] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11694.935259] Key type lgssc registered [11696.225793] Lustre: Echo OBD driver; http://www.lustre.org/ [11737.307801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11737.329961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.684453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11738.723562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11738.820680] Lustre: lustre-MDT0000: new disk, initializing [11738.899779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11738.930673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11744.271363] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11757.243917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.364382] Lustre: 319094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11757.408891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11757.411716] Lustre: Skipped 1 previous similar message [11757.493226] Lustre: lustre-MDT0001: new disk, initializing [11757.605981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11757.643419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11757.663869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11763.343504] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11774.687983] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.087334] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11789.265435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.602271] Lustre: lustre-OST0000: new disk, initializing [11789.608641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11789.621089] Lustre: 321063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11789.743811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11797.645872] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11799.655139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11799.666693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11799.776169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11807.895714] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11810.606378] Lustre: Mounted lustre-client - version 2.17.57_3_g9f8445d [11811.915096] Lustre: Unmounted lustre-client [11825.632316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11825.635214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.644241] LustreError: Skipped 1 previous similar message [11825.648896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11825.662311] Lustre: Skipped 1 previous similar message [11826.161119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11826.173318] Lustre: Skipped 2 previous similar messages [11828.932838] Lustre: server umount lustre-OST0000 complete [11834.337859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11834.342391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.356556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.387990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.388971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.844883] Lustre: server umount lustre-MDT0000 complete [11843.867384] LustreError: 319086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131233 with bad export cookie 353303981606768483 [11843.876034] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11843.880748] LustreError: 319086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11844.690799] Lustre: server umount lustre-MDT0001 complete [11858.041692] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11861.643209] Key type lgssc unregistered [11862.175776] LNet: 1 local NIs in recovery (showing 1): 192.168.202.107@tcp [11863.205249] LNet: 323229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11863.218508] LNetError: 323229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11941.096281] LNet: Removed LNI 192.168.202.107@tcp [11942.217051] Key type .llcrypt unregistered [11942.221273] Key type ._llcrypt unregistered [11962.839202] Key type ._llcrypt registered [11962.841664] Key type .llcrypt registered [11962.946380] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [11980.017314] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11981.002539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11981.028471] alg: No test for adler32 (adler32-zlib) [11982.307867] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11982.728679] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11984.559182] Key type lgssc registered [11986.100757] Lustre: Echo OBD driver; http://www.lustre.org/ [12020.016889] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 05:23:27 (1787131407) [12027.148913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12027.192624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.472958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12028.551710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12028.678909] Lustre: lustre-MDT0000: new disk, initializing [12028.777247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12028.822338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12034.435475] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12045.009830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.164923] Lustre: 326456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12045.218475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12045.223657] Lustre: Skipped 1 previous similar message [12045.287940] Lustre: lustre-MDT0001: new disk, initializing [12045.446990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12045.492859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12045.504484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12049.675538] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12058.773350] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12064.778542] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12070.955027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12071.307552] Lustre: lustre-OST0000: new disk, initializing [12071.321305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12071.333328] Lustre: 328425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12071.453603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12078.224742] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12079.154630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12079.161349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12079.316151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12088.094554] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12110.305772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12110.311459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12110.316195] LustreError: Skipped 1 previous similar message [12110.319038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12110.335595] Lustre: Skipped 1 previous similar message [12113.176400] Lustre: server umount lustre-OST0000 complete [12120.544184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12120.549741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12120.560063] Lustre: Skipped 1 previous similar message [12120.588793] Lustre: Skipped 2 previous similar messages [12124.383534] Lustre: server umount lustre-MDT0000 complete [12129.173047] LustreError: 327260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131519 with bad export cookie 12427895377581806309 [12129.189354] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12129.652390] Lustre: server umount lustre-MDT0001 complete [12143.077211] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12146.067939] Key type lgssc unregistered [12146.407864] LNet: 330338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12146.413338] LNetError: 330338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12147.435744] LNet: Removed LNI 192.168.202.107@tcp [12148.809522] Key type .llcrypt unregistered [12148.811560] Key type ._llcrypt unregistered [12174.394152] Key type ._llcrypt registered [12174.395715] Key type .llcrypt registered [12174.633995] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12177.536594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12177.572598] alg: No test for adler32 (adler32-zlib) [12178.853711] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12179.335804] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12181.087204] Key type lgssc registered [12182.655548] Lustre: Echo OBD driver; http://www.lustre.org/ [12192.049061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12192.081042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.894383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12199.328811] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12203.490433] LustreError: 331613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.519049] LustreError: 331613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12208.041267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.696245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12214.308809] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12223.126249] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12230.295971] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12237.497265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12237.968652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12244.922748] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12253.494332] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12264.202834] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12265.668861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12270.048284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12270.060798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.079670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12271.511177] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [12274.144800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12274.158572] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12276.602678] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [12276.611098] Lustre: Skipped 2 previous similar messages [12281.724112] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [12281.732983] Lustre: Skipped 3 previous similar messages [12284.388899] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12284.847370] Lustre: server umount lustre-MDT0000 complete [12286.848565] LustreError: 331610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12288.415708] LustreError: 331593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131678 with bad export cookie 7471101977708403636 [12288.419765] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12288.425571] LustreError: 331593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12289.507737] LustreError: 332902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.508501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12289.540276] Lustre: Skipped 1 previous similar message [12291.978192] LustreError: 331609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12292.000352] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [12292.016245] Lustre: Skipped 4 previous similar messages [12294.956461] Lustre: server umount lustre-MDT0001 complete [12298.879975] Lustre: server umount lustre-OST0000 complete [12308.940812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.403407] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12317.791806] Lustre: 335800:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.107@tcp [12320.241352] Lustre: server umount MGS complete [12330.324731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12330.884862] LustreError: 336370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.905390] LustreError: 336370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12331.008810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.640371] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12340.198614] LustreError: 336371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.222491] LustreError: 336371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12347.263358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12347.773763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12353.155858] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12363.248785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.588514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12371.852890] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12380.259957] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12382.575991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12389.206852] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12391.204918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12397.101745] Lustre: server umount lustre-OST0000 complete [12402.658603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12402.668583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12402.679201] Lustre: Skipped 1 previous similar message [12402.683329] Lustre: Skipped 1 previous similar message [12412.078906] Lustre: server umount lustre-MDT0000 complete [12412.896770] LustreError: 336371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.930209] LustreError: 336371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12413.919615] Lustre: 330926:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787131787/real 1787131787] req@00000000fab08ee5 x1873943272160512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787131803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12413.945586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12415.012681] Lustre: 330925:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787131788/real 1787131788] req@00000000be8d33ab x1873943272160896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787131804 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12416.793074] LustreError: 336352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131806 with bad export cookie 7471101977708404469 [12416.817826] LustreError: 336352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12416.821042] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12417.409215] Lustre: server umount lustre-MDT0001 complete [12428.390368] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12431.337730] Key type lgssc unregistered [12431.670746] LNet: 339809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12431.698167] LNetError: 339809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12431.719262] LNet: Removed LNI 192.168.202.107@tcp [12432.706518] Key type .llcrypt unregistered [12432.707973] Key type ._llcrypt unregistered [12455.462608] Key type ._llcrypt registered [12455.464955] Key type .llcrypt registered [12455.558203] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12469.580855] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12471.136264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12471.281602] alg: No test for adler32 (adler32-zlib) [12472.457401] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12472.752104] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12474.487312] Key type lgssc registered [12476.379831] Lustre: Echo OBD driver; http://www.lustre.org/ [12515.770699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12515.790497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12517.136890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12517.187512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12517.295673] Lustre: lustre-MDT0000: new disk, initializing [12517.429455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12517.449649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12522.181938] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12533.228355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12533.423838] Lustre: 342971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12533.490926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12533.499777] Lustre: Skipped 1 previous similar message [12533.639311] Lustre: lustre-MDT0001: new disk, initializing [12533.755981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12533.797765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12533.815434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12538.684124] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12548.635585] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12554.764823] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12562.871255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12563.202670] Lustre: lustre-OST0000: new disk, initializing [12563.205528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12563.211254] Lustre: 344940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12563.294759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12569.945091] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12571.249620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12571.254175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12571.349195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12579.608121] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12584.365504] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12584.663933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12589.968655] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12590.098944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12597.226897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12597.237294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.241115] LustreError: Skipped 1 previous similar message [12597.249509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12597.304813] Lustre: Skipped 1 previous similar message [12599.090738] Lustre: server umount lustre-OST0000 complete [12605.409191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12605.422995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.450234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12605.452658] Lustre: Skipped 1 previous similar message [12607.467200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.476449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12607.506384] Lustre: Skipped 1 previous similar message [12610.792980] Lustre: server umount lustre-MDT0000 complete [12615.873443] LustreError: 343766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132005 with bad export cookie 13525046677068398363 [12615.877048] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12615.889045] LustreError: 343766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12616.709316] Lustre: server umount lustre-MDT0001 complete [12626.958959] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 05:33:34 (1787132014) [12645.133767] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12657.367530] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12700.240125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.486299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12700.507753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12700.564604] Lustre: lustre-MDT0000: new disk, initializing [12700.728996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12700.772743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12706.394553] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12717.674283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.884878] Lustre: 349704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12717.996179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12718.000771] Lustre: Skipped 1 previous similar message [12718.121378] Lustre: lustre-MDT0001: new disk, initializing [12718.240942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12718.307665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12718.316174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12724.394736] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12736.509367] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12743.113081] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12751.007490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12751.370840] Lustre: lustre-OST0000: new disk, initializing [12751.374485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12751.392220] Lustre: 351673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12751.474150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12758.487236] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12759.146651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12759.156750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12759.302645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12768.684773] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12805.618402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12805.633542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12805.634798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12805.656507] Lustre: Skipped 1 previous similar message [12805.670105] Lustre: Skipped 1 previous similar message [12808.675314] Lustre: server umount lustre-OST0000 complete [12815.840384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12815.843594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12815.849101] LustreError: Skipped 1 previous similar message [12815.873731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12819.106512] Lustre: server umount lustre-MDT0000 complete [12820.966607] LustreError: 349712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12820.992046] LustreError: 349712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12823.640985] LustreError: 349697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132213 with bad export cookie 13525046677068399245 [12823.643129] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12823.662698] LustreError: 349697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12824.014966] Lustre: server umount lustre-MDT0001 complete [12838.287809] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12841.019660] Key type lgssc unregistered [12841.351960] LNet: 353829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12841.363505] LNetError: 353829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12841.373529] LNet: Removed LNI 192.168.202.107@tcp [12842.379560] Key type .llcrypt unregistered [12842.382147] Key type ._llcrypt unregistered [12947.957536] Key type ._llcrypt registered [12947.960363] Key type .llcrypt registered [12948.073287] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12948.405323] Key type .llcrypt unregistered [12948.407080] Key type ._llcrypt unregistered [13022.590740] Key type ._llcrypt registered [13022.593570] Key type .llcrypt registered [13022.701613] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13023.096528] Key type .llcrypt unregistered [13023.099579] Key type ._llcrypt unregistered [13057.528183] Key type ._llcrypt registered [13057.533588] Key type .llcrypt registered [13057.741496] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13058.044364] Key type .llcrypt unregistered [13058.050489] Key type ._llcrypt unregistered [13090.814617] Key type ._llcrypt registered [13090.816163] Key type .llcrypt registered [13090.891499] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13091.061909] Key type .llcrypt unregistered [13091.063748] Key type ._llcrypt unregistered [13177.394556] Key type ._llcrypt registered [13177.401194] Key type .llcrypt registered [13177.549080] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13178.035890] Key type .llcrypt unregistered [13178.038608] Key type ._llcrypt unregistered [13210.740434] Key type ._llcrypt registered [13210.742715] Key type .llcrypt registered [13210.893896] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13211.305606] Key type .llcrypt unregistered [13211.309716] Key type ._llcrypt unregistered [13242.363908] Key type ._llcrypt registered [13242.366777] Key type .llcrypt registered [13242.479880] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13242.754346] Key type .llcrypt unregistered [13242.756479] Key type ._llcrypt unregistered [13273.096186] Key type ._llcrypt registered [13273.098823] Key type .llcrypt registered [13273.208799] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13273.473339] Key type .llcrypt unregistered [13273.475487] Key type ._llcrypt unregistered [13301.729466] Key type ._llcrypt registered [13301.731096] Key type .llcrypt registered [13301.814602] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13302.138371] Key type .llcrypt unregistered [13302.143238] Key type ._llcrypt unregistered [13334.975266] Key type ._llcrypt registered [13334.983292] Key type .llcrypt registered [13335.176718] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13335.593561] Key type .llcrypt unregistered [13335.596682] Key type ._llcrypt unregistered [13398.087846] Key type ._llcrypt registered [13398.091780] Key type .llcrypt registered [13398.221884] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13398.621227] Key type .llcrypt unregistered [13398.623395] Key type ._llcrypt unregistered [13427.905857] Key type ._llcrypt registered [13427.912688] Key type .llcrypt registered [13428.112167] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13428.421576] Key type .llcrypt unregistered [13428.424844] Key type ._llcrypt unregistered [13456.949951] Key type ._llcrypt registered [13456.954064] Key type .llcrypt registered [13457.120556] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13457.506753] Key type .llcrypt unregistered [13457.513984] Key type ._llcrypt unregistered [13515.028421] Key type ._llcrypt registered [13515.030370] Key type .llcrypt registered [13515.143798] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [13532.176452] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [13533.188791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13533.234321] alg: No test for adler32 (adler32-zlib) [13534.403551] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13534.636391] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [13536.255645] Key type lgssc registered [13537.344901] Lustre: Echo OBD driver; http://www.lustre.org/ [13575.668812] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13575.712402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13577.040489] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13577.069337] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13577.180716] Lustre: tst250fs-MDT0000: new disk, initializing [13577.286561] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13577.309448] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13581.976385] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13593.232395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13593.399959] Lustre: 374183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13593.521338] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13593.530628] Lustre: Skipped 1 previous similar message [13593.534199] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [13593.653386] Lustre: tst250fs-MDT0001: new disk, initializing [13593.765397] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13593.832177] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13593.851210] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13598.671901] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13603.867393] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13613.495753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13613.878229] Lustre: tst250fs-OST0000: new disk, initializing [13613.882337] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13613.891662] Lustre: 375814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13614.061022] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13618.025434] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13618.045559] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13618.395347] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13623.668446] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13638.011919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13638.353678] Lustre: tst250fs-OST0001: new disk, initializing [13638.357537] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13638.371561] Lustre: 376686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13638.534712] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13646.557038] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13648.038659] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13648.050983] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13648.170490] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13652.713514] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13655.600515] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13655.609554] Lustre: Skipped 2 previous similar messages [13676.000830] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13676.009924] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13676.037172] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13679.078864] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13679.078888] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13679.102808] Lustre: Skipped 2 previous similar messages [13680.664883] Lustre: server umount tst250fs-MDT0000 complete [13684.532129] LustreError: 374175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787133074 with bad export cookie 2184827139938243401 [13684.551698] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13685.194752] Lustre: server umount tst250fs-MDT0001 complete [13698.846823] Lustre: server umount tst250fs-OST0000 complete [13713.086762] Lustre: server umount tst250fs-OST0001 complete [13732.931783] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [13743.272290] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [13777.355874] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 05:52:44 (1787133164) [13779.177772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13796.739419] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [13809.115191] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [13843.229919] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13593 sec ========= 05:53:50 (1787133230) [13845.172996] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:53:52 (1787133232) === [13849.293400] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:53:56 (1787133236) === [13892.479577] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13895.271929] Key type lgssc unregistered [13895.582765] LNet: 385832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13895.589880] LNetError: 385832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13895.613500] LNet: Removed LNI 192.168.202.107@tcp [13896.673367] Key type .llcrypt unregistered [13896.677992] Key type ._llcrypt unregistered