[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520690359 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002305] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006348] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.010018] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021691] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026078] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032385] *** VALIDATE proc *** [ 0.033177] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035201] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038020] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042228] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.044144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045598] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.051942] ... event mask: 000000070000000f [ 0.052282] rcu: Hierarchical SRCU implementation. [ 0.054286] smp: Bringing up secondary CPUs ... [ 0.055401] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.058690] smp: Brought up 1 node, 4 CPUs [ 0.059914] smpboot: Max logical packages: 1 [ 0.060008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132395] node 0 deferred pages initialised in 69ms [ 0.135196] devtmpfs: initialized [ 0.136194] x86/mm: Memory block size: 128MB [ 0.137739] gcov: version magic: 0x41383552 [ 0.141192] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146284] pinctrl core: initialized pinctrl subsystem [ 0.148151] [ 0.148612] ************************************************************* [ 0.150012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.154011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.159012] ** This means that this kernel is built to expose internal ** [ 0.162012] ** IOMMU data structures, which may compromise security on ** [ 0.164012] ** your system. ** [ 0.167013] ** ** [ 0.169015] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.173096] ** ** [ 0.176013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178012] ************************************************************* [ 0.181797] NET: Registered protocol family 16 [ 0.183506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.186057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192036] cpuidle: using governor menu [ 0.193560] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196619] PCI: Using configuration type 1 for base access [ 0.199142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215077] cryptd: max_cpu_qlen set to 1000 [ 0.218250] ACPI: Added _OSI(Module Device) [ 0.219017] ACPI: Added _OSI(Processor Device) [ 0.221014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227055] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233288] ACPI: Interpreter enabled [ 0.234061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236019] ACPI: Using IOAPIC for interrupt routing [ 0.238136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251629] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266000] acpiphp: Slot [2] registered [ 0.267116] acpiphp: Slot [5] registered [ 0.269123] acpiphp: Slot [6] registered [ 0.270084] acpiphp: Slot [7] registered [ 0.272130] acpiphp: Slot [8] registered [ 0.274152] acpiphp: Slot [9] registered [ 0.275122] acpiphp: Slot [10] registered [ 0.278133] acpiphp: Slot [3] registered [ 0.279099] acpiphp: Slot [4] registered [ 0.281092] acpiphp: Slot [11] registered [ 0.282099] acpiphp: Slot [12] registered [ 0.284100] acpiphp: Slot [13] registered [ 0.285120] acpiphp: Slot [14] registered [ 0.287094] acpiphp: Slot [15] registered [ 0.288153] acpiphp: Slot [16] registered [ 0.290102] acpiphp: Slot [17] registered [ 0.291086] acpiphp: Slot [18] registered [ 0.292116] acpiphp: Slot [19] registered [ 0.294211] acpiphp: Slot [20] registered [ 0.296119] acpiphp: Slot [21] registered [ 0.297101] acpiphp: Slot [22] registered [ 0.299090] acpiphp: Slot [23] registered [ 0.300106] acpiphp: Slot [24] registered [ 0.302094] acpiphp: Slot [25] registered [ 0.303098] acpiphp: Slot [26] registered [ 0.304067] acpiphp: Slot [27] registered [ 0.305075] acpiphp: Slot [28] registered [ 0.306095] acpiphp: Slot [29] registered [ 0.307064] acpiphp: Slot [30] registered [ 0.308048] acpiphp: Slot [31] registered [ 0.310066] PCI host bridge to bus 0000:00 [ 0.311017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329295] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333530] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342806] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357652] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363097] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366030] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397346] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.469019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.483555] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.488012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522429] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.559015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572587] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.582018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625989] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.690817] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.700016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.712018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.729016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.739862] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.742382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.744533] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.747385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.749238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.754027] iommu: Default domain type: Passthrough [ 0.755396] SCSI subsystem initialized [ 0.756126] ACPI: bus type USB registered [ 0.758150] usbcore: registered new interface driver usbfs [ 0.759101] usbcore: registered new interface driver hub [ 0.761094] usbcore: registered new device driver usb [ 0.763161] pps_core: LinuxPPS API ver. 1 registered [ 0.765011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.769031] PTP clock support registered [ 0.771106] EDAC MC: Ver: 3.0.0 [ 0.772469] PCI: Using ACPI for IRQ routing [ 0.775023] NetLabel: Initializing [ 0.775908] NetLabel: domain hash size = 128 [ 0.777008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.778059] NetLabel: unlabeled traffic allowed by default [ 0.779100] vgaarb: loaded [ 0.781101] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.782012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.790000] clocksource: Switched to clocksource kvm-clock [ 0.895192] VFS: Disk quotas dquot_6.6.0 [ 0.896552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898720] *** VALIDATE ramfs *** [ 0.899846] *** VALIDATE hugetlbfs *** [ 0.901244] pnp: PnP ACPI init [ 0.903404] pnp: PnP ACPI: found 6 devices [ 0.929537] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932651] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.934699] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.936593] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.938714] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.940967] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.943413] NET: Registered protocol family 2 [ 0.945705] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.950300] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.953611] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.958704] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.962081] TCP: Hash tables configured (established 65536 bind 65536) [ 0.964615] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.967459] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972648] NET: Registered protocol family 1 [ 0.975254] RPC: Registered named UNIX socket transport module. [ 0.980023] RPC: Registered udp transport module. [ 0.981603] RPC: Registered tcp transport module. [ 0.983537] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986158] NET: Registered protocol family 44 [ 0.988225] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990528] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992906] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995482] PCI: CLS 0 bytes, default 64 [ 0.997249] Unpacking initramfs... [ 2.341454] debug: unmapping init [mem 0xffff971ffcc54000-0xffff971ffffbffff] [ 2.343824] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.346023] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.348705] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.830629] Initialise system trusted keyrings [ 2.832149] Key type blacklist registered [ 2.834466] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.840798] zbud: loaded [ 2.842885] *** VALIDATE nfs *** [ 2.843901] *** VALIDATE nfs4 *** [ 2.845831] pstore: using deflate compression [ 2.848114] Platform Keyring initialized [ 2.939091] NET: Registered protocol family 38 [ 2.940879] Key type asymmetric registered [ 2.942488] Asymmetric key parser 'x509' registered [ 2.943747] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.945763] io scheduler mq-deadline registered [ 2.946858] io scheduler kyber registered [ 2.948615] io scheduler bfq registered [ 2.949794] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.952670] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.955870] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.958073] ACPI: Power Button [PWRF] [ 2.962634] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.967590] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.987635] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.996048] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.012289] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.040533] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.066914] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.071867] Non-volatile memory driver v1.3 [ 3.073080] Linux agpgart interface v0.103 [ 3.108078] virtio_blk virtio1: [vda] 146136 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.110491] vda: detected capacity change from 0 to 74821632 [ 3.128563] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.131811] vdb: detected capacity change from 0 to 1073741824 [ 3.150945] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153267] vdc: detected capacity change from 0 to 2621440000 [ 3.165957] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.167673] vdd: detected capacity change from 0 to 2621440000 [ 3.180438] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.182881] vde: detected capacity change from 0 to 4294967296 [ 3.203481] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.205932] vdf: detected capacity change from 0 to 4294967296 [ 3.212849] libphy: Fixed MDIO Bus: probed [ 3.219345] usbcore: registered new interface driver usbserial_generic [ 3.221505] usbserial: USB Serial support registered for generic [ 3.223732] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.227487] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.229270] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.231361] mousedev: PS/2 mouse device common for all mice [ 3.233752] rtc_cmos 00:05: RTC can wake from S4 [ 3.236780] rtc_cmos 00:05: registered as rtc0 [ 3.236880] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.238118] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.244688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.245033] intel_pstate: CPU model not supported [ 3.251508] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.252294] hid: raw HID events driver (C) Jiri Kosina [ 3.256854] usbcore: registered new interface driver usbhid [ 3.258242] usbhid: USB HID core driver [ 3.259322] drop_monitor: Initializing network drop monitor service [ 3.260886] Initializing XFRM netlink socket [ 3.262384] NET: Registered protocol family 10 [ 3.264861] Segment Routing with IPv6 [ 3.266390] NET: Registered protocol family 17 [ 3.268538] mpls_gso: MPLS GSO support [ 3.272616] RAS: Correctable Errors collector initialized. [ 3.274967] AVX version of gcm_enc/dec engaged. [ 3.276664] AES CTR mode by8 optimization enabled [ 3.337789] sched_clock: Marking stable (3337770373, 0)->(4237916860, -900146487) [ 3.340443] registered taskstats version 1 [ 3.342196] Loading compiled-in X.509 certificates [ 3.343502] zswap: loaded using pool lzo/zbud [ 3.363400] Key type big_key registered [ 3.371781] Key type encrypted registered [ 3.373452] ima: No TPM chip found, activating TPM-bypass! [ 3.375676] ima: Allocated hash algorithm: sha1 [ 3.377513] ima: No architecture policies found [ 3.379527] evm: Initialising EVM extended attributes: [ 3.381528] evm: security.selinux [ 3.382914] evm: security.ima [ 3.384039] evm: security.capability [ 3.385500] evm: HMAC attrs: 0x1 [ 3.387868] rtc_cmos 00:05: setting system clock to 2026-08-23 09:12:26 UTC (1787476346) [ 3.394233] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.397484] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.408112] Write protecting the kernel read-only data: 28672k [ 3.411478] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.414135] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.446442] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.455643] systemd[1]: Detected virtualization kvm. [ 3.457737] systemd[1]: Detected architecture x86-64. [ 3.459629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.483300] systemd[1]: No hostname configured. [ 3.485222] systemd[1]: Set hostname to . [ 3.487268] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489856] systemd[1]: Initializing machine ID from random generator. [ 3.523886] random: ln: uninitialized urandom read (6 bytes read) [ 3.604698] random: systemd: uninitialized urandom read (16 bytes read) [ 3.607644] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.612937] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.617191] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.118865] device-mapper: uevent: version 1.0.3 [ 4.120279] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [ 4.682688] random: fast init done ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.760860] virtio_net virtio0 ens2: renamed from eth0 [ 4.799561] scsi host0: ata_piix [ 4.822573] scsi host1: ata_piix [ 4.824203] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.826748] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.324458] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.891552] random: crng init done [ 9.892386] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.069675] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.189426] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.454514] SELinux: Disabled at runtime. [ 11.506806] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.512920] systemd[1]: Detected virtualization kvm. [ 11.514718] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.904571] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.907795] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.912906] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.918309] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.921237] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.927843] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.931531] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK [[ 11.995116] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.331441] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.653980] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.681796] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.787697] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.796587] EDAC sbridge: Ver: 1.1.2 [ 14.115563] Key type dns_resolver registered [ 14.408261] NFS: Registering the id_resolver key type [ 14.409503] Key type id_resolver registered [ 14.410564] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 39.685487] libcfs: loading out-of-tree module taints kernel. [ 39.705910] Key type ._llcrypt registered [ 39.707166] Key type .llcrypt registered [ 39.755757] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 48.487696] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 49.107964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.115433] alg: No test for adler32 (adler32-zlib) [ 50.175071] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 50.562383] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 52.207168] Key type lgssc registered [ 52.901896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 60.644412] hrtimer: interrupt took 5392064 ns [ 60.980882] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.053684] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 117.566595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 117.630395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 119.064763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 119.115800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 119.234414] Lustre: lustre-MDT0000: new disk, initializing [ 119.374958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 119.419416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 124.163267] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.895423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.995358] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 139.030430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 139.033456] Lustre: Skipped 1 previous similar message [ 139.157633] Lustre: lustre-MDT0001: new disk, initializing [ 139.251743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.294901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 139.307734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 143.989560] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.804716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 157.364489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 157.790772] Lustre: lustre-OST0000: new disk, initializing [ 157.799397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 157.810162] Lustre: 8430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 157.901525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.194561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.211359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.373651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 163.978472] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.575648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.746906] Lustre: lustre-OST0001: new disk, initializing [ 177.751966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 177.757576] Lustre: 9502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.811206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.328955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 183.340252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 183.384105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 184.148779] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.186950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 205.628528] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 211.509653] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 215.973569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 219.813533] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 221.805934] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 224.150863] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 225.712407] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 05:16:07 EDT 2026 [ 242.601409] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 244.317138] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 262.112546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.123948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.146874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.184241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.185395] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.189236] Lustre: Skipped 2 previous similar messages [ 265.219208] Lustre: Skipped 2 previous similar messages [ 267.647130] Lustre: server umount lustre-MDT0000 complete [ 271.511185] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476614 with bad export cookie 14870188962989034930 [ 271.524461] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.526428] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 275.431171] LustreError: 8437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.432901] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.435974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.435984] Lustre: Skipped 1 previous similar message [ 275.441583] LustreError: 8437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 275.472135] Lustre: Skipped 1 previous similar message [ 276.449843] LustreError: 6499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.480454] LustreError: 6499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 277.991730] Lustre: server umount lustre-MDT0001 complete [ 282.264412] Lustre: server umount lustre-OST0000 complete [ 285.853334] Lustre: server umount lustre-OST0001 complete [ 292.087708] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 304.627707] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 333.912159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.142149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 334.162390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 334.258268] Lustre: lustre-MDT0000: new disk, initializing [ 334.343176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 334.359795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 338.867467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 348.477822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.551746] Lustre: 15058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 348.606954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 348.621229] Lustre: Skipped 1 previous similar message [ 348.748994] Lustre: lustre-MDT0001: new disk, initializing [ 348.875627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 348.899818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 348.909815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 353.423921] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 363.235965] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 368.400358] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 375.082722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 375.399886] Lustre: lustre-OST0000: new disk, initializing [ 375.402935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 375.409783] Lustre: 17028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 375.484913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.953521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 376.965383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 377.094938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 381.387410] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 391.511953] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 397.796362] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 398.087582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 403.505069] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 403.830076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 408.032675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.035619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.049541] LustreError: Skipped 1 previous similar message [ 408.056171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 408.059991] Lustre: Skipped 1 previous similar message [ 408.070678] Lustre: Skipped 4 previous similar messages [ 413.156755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.319440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 420.403208] Lustre: server umount lustre-OST0000 complete [ 425.955616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.972968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.998848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.017170] Lustre: Skipped 3 previous similar messages [ 438.239206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 438.332903] LustreError: 18568:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff972051242b00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 438.411276] Lustre: server umount lustre-MDT0000 complete [ 438.753138] LustreError: 16594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.779670] LustreError: 16594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 441.526925] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476784 with bad export cookie 14870188962989036463 [ 441.534334] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.538292] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 441.906355] Lustre: server umount lustre-MDT0001 complete [ 446.130480] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 05:19:48 (1787476788) [ 464.376825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.824954] LustreError: 19904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.911310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.169599] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 469.997183] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.319155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.448700] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 488.085533] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.983583] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.859786] Lustre: Failing over lustre-MDT0000 [ 495.004647] Lustre: server umount lustre-MDT0000 complete [ 498.153523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.163065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.172466] Lustre: Skipped 1 previous similar message [ 498.178139] LustreError: 19904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.189906] LustreError: 19904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 512.124690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.258690] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.382979] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 517.606794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 517.619229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 517.660052] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 526.249463] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 529.705934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.714043] Lustre: Skipped 2 previous similar messages [ 529.718317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.734510] Lustre: Skipped 5 previous similar messages [ 536.303403] Lustre: server umount lustre-MDT0000 complete [ 540.145738] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.166627] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 540.381260] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476883 with bad export cookie 14870188962989037877 [ 540.384594] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.397734] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 540.903553] Lustre: server umount lustre-MDT0001 complete [ 550.442694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.914112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.918761] Lustre: Skipped 2 previous similar messages [ 554.745382] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 563.176405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.676942] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 576.070413] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.994411] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.418215] Lustre: Failing over lustre-MDT0000 [ 584.669620] Lustre: server umount lustre-MDT0000 complete [ 585.195425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.216116] LustreError: 23239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.244838] LustreError: 23239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 601.375239] Lustre: 3629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787476928/real 1787476928] req@ffff97204861e680 x1874304846456192/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787476944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 601.405705] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.133715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.813525] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xce5d861381e55cdf [ 611.844478] Lustre: MGC192.168.202.132@tcp: Connection restored to 0@lo (at 0@lo) [ 611.850777] Lustre: Skipped 1 previous similar message [ 612.293664] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.314250] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 617.450271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.454500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 617.483582] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.535405] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.804740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.811335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.812436] Lustre: Skipped 3 previous similar messages [ 635.614443] Lustre: server umount lustre-MDT0000 complete [ 639.135708] LustreError: 23219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476982 with bad export cookie 14870188962989038815 [ 639.141489] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.149817] LustreError: 23219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 639.564682] Lustre: server umount lustre-MDT0001 complete [ 648.265789] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:23:10 (1787476990) [ 664.201576] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 675.443104] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 718.826540] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 731.615599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.894530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 731.931955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 731.993513] Lustre: lustre-MDT0000: new disk, initializing [ 732.101369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.109421] Lustre: Skipped 2 previous similar messages [ 732.126435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 737.507690] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 748.620625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.760482] Lustre: 29616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 749.272153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 754.818600] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 764.827247] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 770.528333] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 777.461895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.754678] Lustre: 31587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 779.374394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 779.431856] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 779.492796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 779.520388] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 784.291537] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 793.840520] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 801.337967] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 825.828875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.842982] Lustre: Skipped 4 previous similar messages [ 825.847402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.848948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.861079] Lustre: Skipped 3 previous similar messages [ 829.691552] Lustre: server umount lustre-OST0000 complete [ 840.714097] Lustre: server umount lustre-MDT0000 complete [ 841.185742] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.201583] LustreError: 29624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 844.245659] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477187 with bad export cookie 14870188962989039221 [ 844.250975] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.258208] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 844.601355] Lustre: server umount lustre-MDT0001 complete [ 856.392588] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 860.199694] Key type lgssc unregistered [ 860.545345] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.557287] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.581226] LNet: Removed LNI 192.168.202.132@tcp [ 861.639169] Key type .llcrypt unregistered [ 861.644311] Key type ._llcrypt unregistered [ 891.743946] Key type ._llcrypt registered [ 891.746595] Key type .llcrypt registered [ 891.919424] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 893.444301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.528230] alg: No test for adler32 (adler32-zlib) [ 894.825529] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 895.194566] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 896.880052] Key type lgssc registered [ 898.239799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 906.951818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 906.971966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.495056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.711295] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 917.991981] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.042141] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 922.041682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.497885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.544961] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 935.295728] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 943.405713] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 952.589341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.092580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.185768] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 966.491906] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 967.423292] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 982.465357] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:28:43 (1787477323) [ 984.971412] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 986.847211] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 997.071388] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:28:58 (1787477338) [ 1007.740656] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:29:09 (1787477349) [ 1008.873324] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1019.158684] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1019.169378] Lustre: Skipped 2 previous similar messages [ 1021.813178] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:29:22 (1787477362) [ 1023.632938] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1023.637245] Lustre: Skipped 2 previous similar messages [ 1032.167487] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1032.176049] Lustre: Skipped 4 previous similar messages [ 1047.543533] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:29:48 (1787477388) [ 1050.156984] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1054.040261] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1062.837759] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:30:04 (1787477404) [ 1064.967245] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1064.969857] Lustre: Skipped 6 previous similar messages [ 1078.949528] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:30:20 (1787477420) [ 1095.651078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1095.663250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.674158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.682936] Lustre: Skipped 1 previous similar message [ 1095.695839] Lustre: Skipped 1 previous similar message [ 1099.687296] Lustre: server umount lustre-OST0000 complete [ 1105.899346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.912941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.920396] Lustre: Skipped 1 previous similar message [ 1105.938646] Lustre: Skipped 1 previous similar message [ 1109.910466] Lustre: server umount lustre-MDT0000 complete [ 1111.016461] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.034080] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1113.971050] LustreError: 38898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477457 with bad export cookie 10319455359979153831 [ 1113.982160] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.998302] LustreError: 38898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1114.602644] Lustre: server umount lustre-MDT0001 complete [ 1126.639671] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1129.171401] Key type lgssc unregistered [ 1129.451173] LNet: 42848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.459804] LNetError: 42848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.496041] LNet: Removed LNI 192.168.202.132@tcp [ 1130.229380] Key type .llcrypt unregistered [ 1130.231724] Key type ._llcrypt unregistered [ 1148.126954] Key type ._llcrypt registered [ 1148.137197] Key type .llcrypt registered [ 1148.234939] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 1164.029768] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1165.271108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.323682] alg: No test for adler32 (adler32-zlib) [ 1166.345409] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1166.569465] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1168.264925] Key type lgssc registered [ 1169.380745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.300785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.327838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.622562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1208.652865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1208.768395] Lustre: lustre-MDT0000: new disk, initializing [ 1208.903760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.949768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1213.796452] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1225.308716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.445488] Lustre: 46001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1225.495471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1225.498899] Lustre: Skipped 1 previous similar message [ 1225.561604] Lustre: lustre-MDT0001: new disk, initializing [ 1225.666572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1225.702888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1225.716128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1231.177438] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1241.306685] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1247.776900] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.273717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.474627] Lustre: lustre-OST0000: new disk, initializing [ 1255.477494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1255.483630] Lustre: 47969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1255.553783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.305910] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1265.209491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1265.221862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1265.331518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1270.186343] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.735473] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1276.062418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1281.389914] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1281.666233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1286.117196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.126764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.147506] Lustre: Skipped 1 previous similar message [ 1286.156262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.171387] Lustre: Skipped 1 previous similar message [ 1289.716911] Lustre: server umount lustre-OST0000 complete [ 1296.379256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.406081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.419575] Lustre: Skipped 1 previous similar message [ 1301.164785] Lustre: server umount lustre-MDT0000 complete [ 1306.595194] LustreError: 46014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.644337] LustreError: 46014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1306.696388] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477649 with bad export cookie 18179279799270064757 [ 1306.718745] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1306.729628] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.448535] Lustre: server umount lustre-MDT0001 complete [ 1320.499615] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:34:22 (1787477662) [ 1327.842969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.348307] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.394374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.029837] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1333.732217] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.864026] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.710959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.099441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1347.597305] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1356.245111] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.671899] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.554312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.908746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.452553] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1387.723867] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1392.538320] LustreError: 52966:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1393.463472] Lustre: Modifying parameter general.timeout=1 in log params [ 1394.618739] Lustre: Modifying parameter general.timeout=2 in log params [ 1397.611339] Lustre: Modifying parameter general.timeout=5 in log params [ 1397.616285] Lustre: Skipped 2 previous similar messages [ 1402.360311] Lustre: Modifying parameter general.timeout=10 in log params [ 1402.365470] Lustre: Skipped 4 previous similar messages [ 1410.936287] Lustre: Modifying parameter general.timeout=19 in log params [ 1410.940762] Lustre: Skipped 8 previous similar messages [ 1427.021797] Lustre: Modifying parameter general.timeout=40 in log params [ 1427.025589] Lustre: Skipped 20 previous similar messages [ 1459.848930] Lustre: Modifying parameter general.timeout=74 in log params [ 1459.854947] Lustre: Skipped 33 previous similar messages [ 1519.464368] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:37:41 (1787477861) [ 1521.912707] Lustre: 59903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1523.209409] Lustre: 60002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1523.970760] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1523.982751] Lustre: Skipped 59 previous similar messages [ 1526.586504] Lustre: 60211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1526.602768] Lustre: 60211:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1530.675481] Lustre: 60409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1530.693955] Lustre: 60409:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1559.584596] Lustre: 61785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1559.603556] LustreError: 61785:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1563.313900] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:38:24 (1787477904) [ 1577.470764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.472471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.482969] Lustre: Skipped 1 previous similar message [ 1577.484696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.192251] Lustre: server umount lustre-MDT0000 complete [ 1582.561034] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.585203] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1584.473960] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477927 with bad export cookie 18179279799270065639 [ 1584.478028] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1584.482466] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1584.686976] Lustre: server umount lustre-MDT0001 complete [ 1598.660773] Lustre: server umount lustre-OST0000 complete [ 1623.868986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.287576] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1624.763178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.200452] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1637.448956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.527725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1637.554389] Lustre: 64316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1642.581359] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1649.841708] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1655.138982] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1661.442600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1661.603860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1661.903859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.911448] Lustre: Skipped 1 previous similar message [ 1668.002466] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1674.968887] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.125605] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1676.131666] Lustre: Skipped 19 previous similar messages [ 1676.159545] Lustre: 66881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1681.888949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.897440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.909712] Lustre: Skipped 2 previous similar messages [ 1681.920074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.928666] Lustre: Skipped 2 previous similar messages [ 1687.473190] Lustre: server umount lustre-OST0000 complete [ 1692.644834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.658163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.664072] Lustre: Skipped 2 previous similar messages [ 1692.682442] Lustre: Skipped 3 previous similar messages [ 1698.208617] Lustre: server umount lustre-MDT0000 complete [ 1701.917315] LustreError: 64308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478045 with bad export cookie 18179279799270068621 [ 1701.925088] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.934859] LustreError: 64308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1702.502957] Lustre: server umount lustre-MDT0001 complete [ 1713.395956] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1716.503817] Key type lgssc unregistered [ 1716.776826] LNet: 68090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.779930] LNetError: 68090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.801491] LNet: Removed LNI 192.168.202.132@tcp [ 1718.403270] Key type .llcrypt unregistered [ 1718.408207] Key type ._llcrypt unregistered [ 1746.791288] Key type ._llcrypt registered [ 1746.795152] Key type .llcrypt registered [ 1746.899039] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1748.061233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1748.084806] alg: No test for adler32 (adler32-zlib) [ 1749.182227] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1749.637851] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1751.401191] Key type lgssc registered [ 1752.556767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1760.352792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1760.370560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.172309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.988073] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1772.513515] LustreError: 69506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.538556] LustreError: 69506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1777.664884] LustreError: 69505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.752950] LustreError: 69506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.878972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.333479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1790.899695] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1798.923550] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1805.457508] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1813.863227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.587241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.073783] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1828.540052] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1840.878915] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1850.336395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1850.344753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.356942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.410371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.412431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.448432] Lustre: Skipped 2 previous similar messages [ 1855.501702] Lustre: server umount lustre-MDT0000 complete [ 1858.528874] LustreError: 70237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.548392] LustreError: 70237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1859.084606] LustreError: 69486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478202 with bad export cookie 7386388336788599446 [ 1859.096161] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.404407] Lustre: server umount lustre-MDT0001 complete [ 1873.263759] Lustre: server umount lustre-OST0000 complete [ 1906.517377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.941473] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1906.985390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1907.443098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.111128] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1922.667522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.828329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1922.893617] Lustre: 74379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1923.417701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.371592] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1934.469776] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.885098] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.492335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1946.782990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1947.392551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.228289] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1962.855505] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1964.352574] Lustre: 76955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1965.677632] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1974.753419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.764088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.781740] Lustre: Skipped 1 previous similar message [ 1974.785960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.555954] Lustre: server umount lustre-OST0000 complete [ 1984.481030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1984.498367] LustreError: Skipped 1 previous similar message [ 1984.502804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.530014] Lustre: Skipped 1 previous similar message [ 1984.545843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.562713] Lustre: Skipped 2 previous similar messages [ 1986.937946] Lustre: server umount lustre-MDT0000 complete [ 1990.578338] LustreError: 74373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478333 with bad export cookie 7386388336788600300 [ 1990.582956] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.584921] LustreError: 74373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1990.959208] Lustre: server umount lustre-MDT0001 complete [ 1999.577674] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2002.004163] Key type lgssc unregistered [ 2002.242769] LNet: 78167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.248388] LNetError: 78167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.264978] LNet: Removed LNI 192.168.202.132@tcp [ 2002.980967] Key type .llcrypt unregistered [ 2002.983042] Key type ._llcrypt unregistered [ 2031.865188] Key type ._llcrypt registered [ 2031.867390] Key type .llcrypt registered [ 2032.022073] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2033.682975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.710059] alg: No test for adler32 (adler32-zlib) [ 2034.950653] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2035.442532] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2037.215179] Key type lgssc registered [ 2039.374168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2047.594681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2047.617338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.161393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.352863] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2059.748580] LustreError: 79581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.761667] LustreError: 79581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2061.114387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.601432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.511589] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2071.225385] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.298595] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.605452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.171814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.262876] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2097.429994] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2103.986582] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2105.264855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2106.744654] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2109.756721] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2109.759630] Lustre: Skipped 1 previous similar message [ 2114.453929] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2114.469342] Lustre: Skipped 3 previous similar messages [ 2123.059216] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2123.072506] Lustre: Skipped 9 previous similar messages [ 2139.304798] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2139.308117] Lustre: Skipped 18 previous similar messages [ 2171.558986] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2171.564467] Lustre: Skipped 37 previous similar messages [ 2198.873535] Lustre: 87683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2201.343744] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:49:03 (1787478543) [ 2202.666688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2204.400861] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:49:06 (1787478546) [ 2209.248468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2209.254088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.275538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.322582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.335794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.646918] Lustre: server umount lustre-OST0000 complete [ 2220.002876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2220.013362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.038152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2220.047447] Lustre: Skipped 1 previous similar message [ 2222.562026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.567281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.534553] Lustre: server umount lustre-MDT0000 complete [ 2227.694577] LustreError: 86772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.720502] LustreError: 86772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2227.740393] LustreError: 79563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478570 with bad export cookie 4376346080424130811 [ 2227.748598] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2228.090498] Lustre: server umount lustre-MDT0001 complete [ 2238.176209] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2240.605531] Key type lgssc unregistered [ 2240.924284] LNet: 88935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.933567] LNetError: 88935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.963288] LNet: Removed LNI 192.168.202.132@tcp [ 2242.732885] Key type .llcrypt unregistered [ 2242.735734] Key type ._llcrypt unregistered [ 2247.292728] Key type ._llcrypt registered [ 2247.294268] Key type .llcrypt registered [ 2247.384287] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2251.075453] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules [ 2252.134714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.163561] alg: No test for adler32 (adler32-zlib) [ 2253.271352] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2253.510938] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2255.175164] Key type lgssc registered [ 2256.477354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.530903] LustreError: 90136:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2273.942892] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2279.168871] LustreError: 90136:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2281.436979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2281.456403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.009215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.821393] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2292.205197] LustreError: 91162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.229294] LustreError: 91162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2297.316164] LustreError: 91161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.851261] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:50:39 (1787478639) [ 2301.851185] LustreError: 91193:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2301.867922] LustreError: 91193:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2301.875612] LustreError: 91193:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2302.411745] Lustre: server umount lustre-MDT0000 complete [ 2320.078430] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2324.372852] Key type lgssc unregistered [ 2324.983303] LNet: 92690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.009152] LNetError: 92690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.096250] LNet: Removed LNI 192.168.202.132@tcp [ 2326.392834] Key type .llcrypt unregistered [ 2326.404979] Key type ._llcrypt unregistered [ 2343.556245] Key type ._llcrypt registered [ 2343.557985] Key type .llcrypt registered [ 2343.617503] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2344.179187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.251276] alg: No test for adler32 (adler32-zlib) [ 2345.190648] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2345.410200] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2347.039684] Key type lgssc registered [ 2348.024752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.739117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.766976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.279714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.895655] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2367.458138] LustreError: 93910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.472202] LustreError: 93910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2369.890323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.562365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.447550] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2382.619200] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2389.272673] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.918070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.825173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.125522] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2411.446643] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2468.453568] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:53:29 (1787478809) [ 2470.891584] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2473.263791] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:53:34 (1787478814) [ 2482.153630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.159842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.172833] Lustre: Skipped 2 previous similar messages [ 2482.184920] Lustre: Skipped 2 previous similar messages [ 2485.196807] Lustre: server umount lustre-MDT0000 complete [ 2487.270442] LustreError: 93909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.298355] LustreError: 93909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2489.572996] LustreError: 93890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478832 with bad export cookie 6700862709822124270 [ 2489.586077] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.593341] LustreError: 93890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2490.240846] Lustre: server umount lustre-MDT0001 complete [ 2505.005319] Lustre: server umount lustre-OST0000 complete [ 2518.358594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.333554] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.631154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.648515] LustreError: 98206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.334304] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2529.762592] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.811161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.841251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.204381] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2550.517969] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2556.222755] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2568.705421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.895531] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2568.898615] LustreError: 98193:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2568.914641] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2568.917696] LustreError: 100288:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2568.920366] LustreError: 100288:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2568.924420] LustreError: 100288:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2568.966891] Lustre: server umount lustre-OST0000 complete [ 2568.968199] LustreError: 100288:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2576.945864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.077052] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2577.083721] LustreError: 98195:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2577.103451] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2577.110437] LustreError: 100753:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2577.117269] LustreError: 100753:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2577.121583] LustreError: 100753:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2577.189927] Lustre: server umount lustre-OST0000 complete [ 2577.197666] LustreError: 100753:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2585.675658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.883188] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2585.891053] Lustre: Found index 0 for lustre-OST0000, updating log [ 2585.901492] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2586.012300] Lustre: lustre-OST0000: new disk, initializing [ 2586.014149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2586.017691] Lustre: 101289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2586.279890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.348378] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2597.645038] Lustre: Failing over lustre-OST0000 [ 2597.982901] Lustre: server umount lustre-OST0000 complete [ 2602.976628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.989930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.001085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.029054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.029384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.169409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.182214] Lustre: Skipped 4 previous similar messages [ 2612.263967] Lustre: server umount lustre-MDT0000 complete [ 2616.153971] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478959 with bad export cookie 6700862709822127007 [ 2616.165846] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.167641] LustreError: 98185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2616.435265] Lustre: server umount lustre-MDT0001 complete [ 2625.348790] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:56:07 (1787478967) [ 2640.347744] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2649.824600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.299150] LustreError: 103634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.315508] LustreError: 103634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2650.410027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.818448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2663.798526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.041494] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2671.989687] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2678.379356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.965907] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2684.635811] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2689.209406] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2689.224479] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2689.251408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2692.978393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.064234] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2693.133539] Lustre: lustre-OST0001: new disk, initializing [ 2693.136188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2693.140521] Lustre: 105951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2693.350099] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2693.365337] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.367619] Lustre: Skipped 2 previous similar messages [ 2693.371740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2695.463090] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2695.479936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2695.507475] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2695.508251] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2695.521623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2700.274536] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2700.910995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2700.919733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2701.063465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2707.890447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2714.903950] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2721.249600] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.258142] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.278282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2726.246207] Lustre: server umount lustre-MDT0001 complete [ 2726.888315] LustreError: 105149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.920177] LustreError: 105149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2737.691976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.824519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2737.828679] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2737.871323] Lustre: 103624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.635035] LustreError: 107502:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2739.641375] LustreError: 107502:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2739.672265] LustreError: 103626:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2739.845037] LustreError: 107520:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -17 [ 2739.855587] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2739.855587] [ 2743.808103] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2743.814303] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2753.570808] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:58:15 (1787479095) [ 2771.444569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2789.247979] Lustre: 109568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2809.773297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2812.600961] Lustre: 110702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2891.762720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2891.774358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.790246] Lustre: Skipped 2 previous similar messages [ 2891.797430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.803679] Lustre: Skipped 3 previous similar messages [ 2895.528819] Lustre: server umount lustre-MDT0000 complete [ 2897.387191] LustreError: 103629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.423774] LustreError: 103629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 2899.670704] LustreError: 106672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479242 with bad export cookie 6700862709822128008 [ 2899.676896] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2899.680669] LustreError: 106672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2900.144638] Lustre: server umount lustre-MDT0001 complete [ 2914.306534] Lustre: server umount lustre-OST0000 complete [ 2928.610210] Lustre: server umount lustre-OST0001 complete [ 2934.287366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2945.227705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2952.884973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2962.791284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2982.308780] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2992.521083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.546696] Lustre: lustre-MDT0000: reset Object Index mappings [ 2992.922995] LustreError: 114665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.039690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.044136] Lustre: Skipped 1 previous similar message [ 2997.794097] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3006.852036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.911481] Lustre: lustre-MDT0001: reset Object Index mappings [ 3015.681944] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3018.448312] Lustre: 115812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3025.094635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.919548] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3032.032249] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3038.230238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3042.402834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3048.444483] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3048.455803] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3051.021916] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3059.181802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3062.011862] Lustre: 117691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3108.363806] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 06:04:09 (1787479449) [ 3135.458564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.465406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.468206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.478497] Lustre: Skipped 6 previous similar messages [ 3135.481915] Lustre: Skipped 6 previous similar messages [ 3139.645360] Lustre: server umount lustre-MDT0000 complete [ 3140.578756] LustreError: 114665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.609592] LustreError: 114665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3144.077777] LustreError: 114645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479487 with bad export cookie 6700862709822175090 [ 3144.101647] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.813731] Lustre: server umount lustre-MDT0001 complete [ 3161.271758] Lustre: server umount lustre-OST0000 complete [ 3170.768970] Lustre: server umount lustre-OST0001 complete [ 3178.359748] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3189.994662] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3225.278765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.566724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3225.592456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3225.688639] Lustre: lustre-MDT0000: new disk, initializing [ 3225.804026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.811362] Lustre: Skipped 3 previous similar messages [ 3225.835762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3230.717669] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3237.295281] Lustre: server umount lustre-MDT0000 complete [ 3245.971164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.345813] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3246.362878] Lustre: 121959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3246.379543] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3251.891822] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3257.096290] Lustre: server umount lustre-MDT0000 complete [ 3265.480657] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 06:06:47 (1787479607) [ 3267.115198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3268.749283] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 06:06:50 (1787479610) [ 3282.953712] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3294.785710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.589484] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3314.064364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.176586] Lustre: 124009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3314.243748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3314.252164] Lustre: Skipped 1 previous similar message [ 3314.383670] Lustre: lustre-MDT0001: new disk, initializing [ 3314.612347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3314.635840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3320.250314] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3324.845938] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3330.229600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.459407] Lustre: lustre-OST0000: new disk, initializing [ 3330.462491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3330.467642] Lustre: 125588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3331.939831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3331.948138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3332.071255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3335.742364] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3345.887886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.059317] Lustre: lustre-OST0001: new disk, initializing [ 3346.064307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3346.071034] Lustre: 126456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3347.896423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3347.917291] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3348.030655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3352.136746] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3360.886293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3364.381238] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3713.387331] Lustre: 132861:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3713.408177] Lustre: 132861:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3713.419784] Lustre: 132861:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3713.428347] Lustre: 132861:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3713.440447] Lustre: 132861:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3713.446219] Lustre: 132861:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3994.120959] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:18:54 (1787480334) [ 4011.491884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4011.496995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.511689] Lustre: Skipped 2 previous similar messages [ 4011.517423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.521196] Lustre: Skipped 4 previous similar messages [ 4013.544145] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.563083] Lustre: Skipped 2 previous similar messages [ 4015.832582] Lustre: server umount lustre-MDT0000 complete [ 4018.666039] LustreError: 125581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.683985] LustreError: 125581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4019.784375] LustreError: 132922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480362 with bad export cookie 6700862709822240897 [ 4019.788790] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.794595] LustreError: 132922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4023.799764] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.820297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4023.832780] Lustre: Skipped 5 previous similar messages [ 4026.359210] Lustre: server umount lustre-MDT0001 complete [ 4044.255330] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4044.355863] Lustre: server umount lustre-OST0000 complete [ 4054.736711] Lustre: server umount lustre-OST0001 complete [ 4061.841257] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4072.239736] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4111.766225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.142822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4112.169742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4112.256974] Lustre: lustre-MDT0000: new disk, initializing [ 4112.429165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.434553] Lustre: Skipped 5 previous similar messages [ 4112.472427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4118.585597] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4130.416027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.545798] Lustre: 135915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4130.603151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4130.607848] Lustre: Skipped 1 previous similar message [ 4130.682113] Lustre: lustre-MDT0001: new disk, initializing [ 4130.813527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4130.835921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4135.998375] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4145.580278] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4151.157286] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4157.763860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4158.029151] Lustre: lustre-OST0000: new disk, initializing [ 4158.034517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4158.040652] Lustre: 137885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4158.164710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.174892] Lustre: Skipped 1 previous similar message [ 4160.229401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4160.248993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4160.375569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4165.059515] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4175.028599] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.435397] Lustre: lustre-MDD0000: changelog on [ 4189.730511] Lustre: lustre-MDD0001: changelog on [ 5806.311994] Lustre: lustre-MDD0001: changelog off [ 5808.335512] Lustre: 135923:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5809.305174] Lustre: lustre-MDD0000: changelog off [ 5819.363087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.375770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.389157] LustreError: Skipped 1 previous similar message [ 5819.390155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.390161] Lustre: Skipped 2 previous similar messages [ 5819.427744] Lustre: Skipped 2 previous similar messages [ 5821.276380] Lustre: server umount lustre-OST0000 complete [ 5825.507052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5825.518101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.533482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.541982] Lustre: Skipped 1 previous similar message [ 5829.603629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.618668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.633729] Lustre: server umount lustre-MDT0000 complete [ 5834.721707] LustreError: 135927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.739963] LustreError: 135927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5835.688259] LustreError: 135908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482178 with bad export cookie 6700862709823055242 [ 5835.692664] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.702982] LustreError: 135908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5836.104697] Lustre: server umount lustre-MDT0001 complete [ 5849.119289] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5852.965657] Key type lgssc unregistered [ 5853.310616] LNet: 146997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5853.321508] LNetError: 146997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5853.338282] LNet: Removed LNI 192.168.202.132@tcp [ 5854.648579] Key type .llcrypt unregistered [ 5854.650416] Key type ._llcrypt unregistered [ 5875.571352] Key type ._llcrypt registered [ 5875.573547] Key type .llcrypt registered [ 5875.694517] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5890.612103] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5892.432290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5892.469070] alg: No test for adler32 (adler32-zlib) [ 5893.645368] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 5893.968101] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5895.663232] Key type lgssc registered [ 5897.150747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5934.127573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5934.165570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.345692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5935.367258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5935.423843] Lustre: lustre-MDT0000: new disk, initializing [ 5935.493461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.508582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5940.477586] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5950.504402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.630528] Lustre: 150198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5950.685972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5950.690289] Lustre: Skipped 1 previous similar message [ 5950.761636] Lustre: lustre-MDT0001: new disk, initializing [ 5950.859830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5950.896469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5950.907790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5955.502167] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5964.854396] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5970.291880] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5976.735356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.011437] Lustre: lustre-OST0000: new disk, initializing [ 5977.015823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5977.023348] Lustre: 152170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5977.093956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.393627] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5983.834839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5983.839469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5983.935967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5990.940737] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.405979] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_module obdecho/obdecho [ 5998.331918] LustreError: 153172:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5998.342788] LustreError: 153172:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5999.875176] LustreError: 153268:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6009.829841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.831868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.856786] Lustre: Skipped 1 previous similar message [ 6014.947803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.958473] Lustre: Skipped 4 previous similar messages [ 6015.689566] Lustre: server umount lustre-MDT0000 complete [ 6019.128283] LustreError: 150190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482362 with bad export cookie 18292298702466170880 [ 6019.138212] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6019.147950] LustreError: 150190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6019.789128] Lustre: server umount lustre-MDT0001 complete [ 6034.031479] Lustre: server umount lustre-OST0000 complete [ 6057.092224] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6067.336193] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6102.427099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.659507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6102.688278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6102.772625] Lustre: lustre-MDT0000: new disk, initializing [ 6102.889533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.907862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6107.565900] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6117.903479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.027252] Lustre: 156986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6118.086026] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6118.093942] Lustre: Skipped 1 previous similar message [ 6118.179600] Lustre: lustre-MDT0001: new disk, initializing [ 6118.286182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.336940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6118.347642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6123.230728] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6132.800587] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.160916] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6143.975276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6144.233842] Lustre: lustre-OST0000: new disk, initializing [ 6144.237989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6144.244238] Lustre: 158953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6144.326802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.493879] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6151.726188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6151.738631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6151.841623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6159.379228] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.610355] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6163.798224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6167.921478] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6168.123607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6172.646611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6172.652813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.674185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6172.681338] Lustre: Skipped 1 previous similar message [ 6176.306252] Lustre: server umount lustre-OST0000 complete [ 6182.880448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.884116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6182.899035] Lustre: Skipped 2 previous similar messages [ 6182.926966] Lustre: Skipped 1 previous similar message [ 6186.279882] Lustre: server umount lustre-MDT0000 complete [ 6189.507226] LustreError: 156978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482532 with bad export cookie 18292298702466171839 [ 6189.510471] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6189.514558] LustreError: 156978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6189.790694] Lustre: server umount lustre-MDT0001 complete [ 6193.065235] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:55:35 (1787482535) [ 6194.519227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6196.015612] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:55:37 (1787482537) [ 6202.882151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6203.227142] LustreError: 161363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.249440] LustreError: 161363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6203.342487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.953616] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6208.494168] LustreError: 161364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.615576] LustreError: 161363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.480718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.372971] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6226.063519] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.622675] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6236.476791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.803140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.807918] Lustre: Skipped 1 previous similar message [ 6242.550957] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6249.019341] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.547556] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 6255.585803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6255.607784] LustreError: Skipped 1 previous similar message [ 6255.609123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.453755] Lustre: server umount lustre-MDT0000 complete [ 6258.659374] LustreError: 161363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.680639] LustreError: 161363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6259.828238] LustreError: 161343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482602 with bad export cookie 18292298702466172721 [ 6259.836903] LustreError: 161343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6259.837607] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6263.776154] LustreError: 161359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.776985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.792281] LustreError: 161359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6263.807263] Lustre: Skipped 2 previous similar messages [ 6263.811176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6263.814254] Lustre: Skipped 1 previous similar message [ 6266.241775] Lustre: server umount lustre-MDT0001 complete [ 6272.645934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6284.725345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.966638] LustreError: 165204:0:(llog.c:1655:llog_backup()) MGC192.168.202.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6284.969956] Lustre: 165204:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6285.299948] LustreError: 165209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.315690] LustreError: 165209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6304.737047] LustreError: 165210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.762749] LustreError: 165210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6340.066464] LustreError: 163193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.080262] LustreError: 163193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 6345.695303] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6345.700215] Lustre: 165204:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6386.701282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6390.841440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6391.795126] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6397.848673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.064633] LustreError: 148421:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b0edcebc x1874310974267776/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6404.086540] LustreError: 165211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.100308] LustreError: 165211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6408.122268] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6409.736892] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6414.625258] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6415.974873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6420.340616] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6421.552053] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6435.295935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6435.305624] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.320699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.327342] Lustre: Skipped 1 previous similar message [ 6435.340283] Lustre: Skipped 4 previous similar messages [ 6440.802375] Lustre: server umount lustre-MDT0000 complete [ 6444.018187] LustreError: 161343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482787 with bad export cookie 18292298702466179175 [ 6444.032126] LustreError: 161343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6444.034504] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.358329] Lustre: server umount lustre-MDT0001 complete [ 6457.850446] Lustre: server umount lustre-OST0000 complete [ 6474.828870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6482.773951] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6511.220438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.451436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6511.473413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6511.531368] Lustre: lustre-MDT0000: new disk, initializing [ 6511.593277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6516.007402] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6525.010064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.101706] Lustre: 170305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6525.329630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.339592] Lustre: Skipped 2 previous similar messages [ 6525.380408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6529.123773] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6537.929449] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6542.577078] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.482663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.715183] Lustre: lustre-OST0000: new disk, initializing [ 6548.717519] Lustre: Skipped 1 previous similar message [ 6548.721205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6548.724392] Lustre: Skipped 2 previous similar messages [ 6548.728314] Lustre: 172283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6554.383309] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6557.765153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6557.774152] Lustre: Skipped 1 previous similar message [ 6557.778342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6557.895297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6562.856764] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.157329] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6567.341173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6570.556639] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6570.706627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6573.543477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.546477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6573.556186] Lustre: Skipped 1 previous similar message [ 6573.558228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6573.558238] Lustre: Skipped 3 previous similar messages [ 6586.335120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6586.518199] Lustre: server umount lustre-OST0000 complete [ 6591.972105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6591.977735] LustreError: Skipped 1 previous similar message [ 6597.695850] Lustre: server umount lustre-MDT0000 complete [ 6599.140319] LustreError: 170315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.161313] LustreError: 170315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6601.187722] LustreError: 170297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482944 with bad export cookie 18292298702466196010 [ 6601.197090] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6601.199761] LustreError: 170297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6601.522647] Lustre: server umount lustre-MDT0001 complete [ 6608.766331] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 07:02:30 (1787482950) [ 6613.232710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.084419] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6623.856406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.210453] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6632.421933] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6636.345963] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6640.972805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.971665] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6651.293412] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.983871] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 07:03:31 (1787483011) [ 6674.404984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.405181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6674.407099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6674.412906] Lustre: Skipped 4 previous similar messages [ 6674.430812] Lustre: Skipped 8 previous similar messages [ 6677.734889] Lustre: server umount lustre-OST0000 complete [ 6686.729383] Lustre: server umount lustre-MDT0000 complete [ 6688.942687] LustreError: 174677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483032 with bad export cookie 18292298702466196892 [ 6688.948721] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.952746] LustreError: 174677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6689.148814] Lustre: server umount lustre-MDT0001 complete [ 6695.328363] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6696.900612] Key type lgssc unregistered [ 6697.112509] LNet: 178453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6697.119918] LNetError: 178453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6698.150606] LNet: Removed LNI 192.168.202.132@tcp [ 6698.757857] Key type .llcrypt unregistered [ 6698.759178] Key type ._llcrypt unregistered [ 6711.166406] Key type ._llcrypt registered [ 6711.168171] Key type .llcrypt registered [ 6711.233912] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6711.888349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6711.917050] alg: No test for adler32 (adler32-zlib) [ 6712.968616] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6713.129764] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6714.767168] Key type lgssc registered [ 6715.601738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6720.918628] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6720.931299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6736.415356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6741.543104] LustreError: 179677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 6746.656303] LustreError: 179677:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6746.666596] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6746.672323] LustreError: MGC192.168.202.132@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6746.687281] LustreError: 179677:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6746.695842] LustreError: 179677:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6746.708614] LustreError: 179677:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6746.757984] Lustre: server umount lustre-OST0000 complete [ 6746.760090] LustreError: 179677:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6753.053599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.408334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.980753] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6764.022399] LustreError: 180175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.042187] LustreError: 180175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6764.394075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.691196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6767.893475] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6772.858037] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6776.492126] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6780.993816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6781.189863] LustreError: 181988:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6781.196441] Lustre: 181988:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.132@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6781.311065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.822167] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6791.418761] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.723079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.730260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6794.741019] Lustre: Skipped 1 previous similar message [ 6794.749661] Lustre: Skipped 2 previous similar messages [ 6799.310866] Lustre: server umount lustre-MDT0000 complete [ 6799.842348] LustreError: 181319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6799.862731] LustreError: 181319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6801.921817] LustreError: 180157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483145 with bad export cookie 18199274633497899201 [ 6801.924095] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6801.935583] LustreError: 180157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6802.132259] Lustre: server umount lustre-MDT0001 complete [ 6814.096632] Lustre: server umount lustre-OST0000 complete [ 6821.149182] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6827.955331] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6848.199959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.350670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6848.364414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6848.411592] Lustre: lustre-MDT0000: new disk, initializing [ 6848.460828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.471067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6851.170349] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6858.267927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.368140] Lustre: 185507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6858.406372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6858.409650] Lustre: Skipped 1 previous similar message [ 6858.462970] Lustre: lustre-MDT0001: new disk, initializing [ 6858.512450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6858.530016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6858.533371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6861.089866] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6866.877135] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6870.234721] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6874.357778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6874.522337] Lustre: lustre-OST0000: new disk, initializing [ 6874.524184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6874.527838] Lustre: 187472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6874.579341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.259276] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6883.385640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6883.394962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6883.437328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6884.107683] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.985737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6887.101770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6889.776291] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6889.898967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6893.542067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.542379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.560590] Lustre: Skipped 1 previous similar message [ 6893.564446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6893.569110] Lustre: Skipped 1 previous similar message [ 6897.470437] Lustre: server umount lustre-OST0000 complete [ 6903.783844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.786901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.795388] Lustre: Skipped 2 previous similar messages [ 6903.804708] Lustre: Skipped 1 previous similar message [ 6906.674320] Lustre: server umount lustre-MDT0000 complete [ 6908.896190] LustreError: 185520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.918074] LustreError: 185520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6909.201785] LustreError: 185499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483252 with bad export cookie 18199274633497899922 [ 6909.209852] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.381835] Lustre: server umount lustre-MDT0001 complete [ 6915.110759] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 07:07:37 (1787483257) [ 6927.998533] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6930.144545] Key type lgssc unregistered [ 6930.403761] LNet: 190417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6930.414479] LNetError: 190417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6930.439966] LNet: Removed LNI 192.168.202.132@tcp [ 6931.039575] Key type .llcrypt unregistered [ 6931.041476] Key type ._llcrypt unregistered [ 6943.676599] Key type ._llcrypt registered [ 6943.680144] Key type .llcrypt registered [ 6943.755125] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6944.422170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6944.474379] alg: No test for adler32 (adler32-zlib) [ 6945.479706] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6945.664819] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6947.271234] Key type lgssc registered [ 6948.046197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6953.543478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6953.557103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.926149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.934603] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6964.520685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.769102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6967.849161] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6972.620640] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6975.645479] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6980.540511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.665612] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6980.668667] LustreError: 193476:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6980.674908] Lustre: 193476:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.132@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6980.768341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.675139] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6989.830967] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6994.405087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.407334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.419527] Lustre: Skipped 2 previous similar messages [ 6994.434877] Lustre: Skipped 2 previous similar messages [ 6997.471576] Lustre: server umount lustre-MDT0000 complete [ 6999.524266] LustreError: 192418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.543096] LustreError: 192418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7000.151269] LustreError: 191593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483343 with bad export cookie 726391629350729932 [ 7000.155567] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7000.159448] LustreError: 191593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7000.370466] Lustre: server umount lustre-MDT0001 complete [ 7013.400206] Lustre: server umount lustre-OST0000 complete [ 7019.674203] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7026.025344] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7048.769245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.939409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7048.983912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7049.037637] Lustre: lustre-MDT0000: new disk, initializing [ 7049.101423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.118303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7051.929931] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7058.809114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.876645] Lustre: 196995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7058.900245] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7058.902463] Lustre: Skipped 1 previous similar message [ 7058.904478] LustreError: 197778:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7058.908605] LustreError: 197778:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7058.911988] Lustre: 197778:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7058.922847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7058.925471] Lustre: Skipped 1 previous similar message [ 7058.964575] Lustre: lustre-MDT0001: new disk, initializing [ 7059.035687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7059.066629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7059.078656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7061.818805] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7068.464774] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7072.420766] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.856631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.987992] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7076.990590] Lustre: Skipped 1 previous similar message [ 7076.994558] LustreError: 198939:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7077.000763] LustreError: 198939:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7077.005265] Lustre: 198939:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7077.011225] Lustre: 198939:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7077.024246] Lustre: lustre-OST0000: new disk, initializing [ 7077.027428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7077.031940] Lustre: 198963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7077.080291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.847468] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7085.063498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7085.069932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7085.133811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7086.264126] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7088.791291] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7088.924984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7091.233290] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7091.345140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7095.267157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.269836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7095.272226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7095.285859] LustreError: Skipped 1 previous similar message [ 7098.687239] Lustre: server umount lustre-OST0000 complete [ 7104.993251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7105.002876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.013782] Lustre: Skipped 1 previous similar message [ 7105.016941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7105.020096] Lustre: Skipped 1 previous similar message [ 7106.797985] Lustre: server umount lustre-MDT0000 complete [ 7108.710816] LustreError: 196989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483451 with bad export cookie 726391629350730646 [ 7108.713961] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.718236] LustreError: 196989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.880153] Lustre: server umount lustre-MDT0001 complete [ 7113.569235] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:10:55 (1787483455) [ 7122.651953] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7128.120540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.326744] LustreError: 202084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.353719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.581135] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7133.664514] LustreError: 202085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.899407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.241565] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7138.824816] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7142.340272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7145.682338] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7150.341859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.452562] Lustre: lustre-OST0001: new disk, initializing [ 7150.454932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7150.459807] Lustre: 204389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7152.496786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7152.506770] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7152.540514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7153.913949] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7160.765369] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7167.012860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7167.907382] Lustre: DEBUG MARKER: ADD OST3 [ 7170.220103] loop: module loaded [ 7170.258475] loop0: detected capacity change from 0 to 204800000 [ 7170.421324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7170.427442] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7170.432152] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7173.705159] loop0: detected capacity change from 0 to 204800000 [ 7176.113204] Lustre: DEBUG MARKER: STOP OST3 [ 7177.036508] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.143187] Lustre: lustre-OST0003: new disk, initializing [ 7177.145663] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7177.148944] Lustre: 206225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7177.231212] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7177.234346] Lustre: Skipped 3 previous similar messages [ 7179.229150] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7179.295679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7179.306157] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7179.328348] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7179.627704] Lustre: Failing over lustre-OST0003 [ 7179.715788] Lustre: server umount lustre-OST0003 complete [ 7180.563452] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7181.279166] LustreError: 206562:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7183.309184] LustreError: 203569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.359233] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.364653] Lustre: Skipped 1 previous similar message [ 7184.367123] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.693268] Lustre: DEBUG MARKER: START OST3 again [ 7188.400050] LustreError: 205175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.407602] LustreError: 205175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7188.918442] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.035221] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7190.583486] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7190.844774] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7190.847958] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7191.525707] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7207.290912] Lustre: Failing over lustre-OST0003 [ 7207.343366] Lustre: server umount lustre-OST0003 complete [ 7208.895881] LustreError: 205175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7209.447529] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.452696] Lustre: Skipped 2 previous similar messages [ 7222.239962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7222.246270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.254214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7222.262644] Lustre: Skipped 2 previous similar messages [ 7227.175973] Lustre: server umount lustre-MDT0000 complete [ 7228.750233] LustreError: 202066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483571 with bad export cookie 726391629350731528 [ 7228.754455] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7228.755206] LustreError: 202066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7229.920107] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.922451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7229.927232] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7229.929705] Lustre: Skipped 5 previous similar messages [ 7238.049106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7238.054179] Lustre: Skipped 3 previous similar messages [ 7243.231164] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7243.358079] Lustre: server umount lustre-MDT0001 complete [ 7245.422746] Lustre: server umount lustre-OST0000 complete [ 7247.635556] Lustre: server umount lustre-OST0001 complete [ 7250.704597] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7254.176584] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7270.661508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.770237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7270.785970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7270.831968] Lustre: lustre-MDT0000: new disk, initializing [ 7270.874298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.877218] Lustre: Skipped 1 previous similar message [ 7270.884309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7272.771384] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7278.676157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.747866] Lustre: 211689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7278.894532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7280.844266] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7286.229302] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7288.691100] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7291.818386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7291.945435] Lustre: lustre-OST0000: new disk, initializing [ 7291.948011] Lustre: Skipped 1 previous similar message [ 7291.950570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7291.953365] Lustre: Skipped 2 previous similar messages [ 7291.955826] Lustre: 213656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7293.203776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7293.207982] Lustre: Skipped 1 previous similar message [ 7293.217558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7293.269659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7295.094621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7302.664511] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7308.034230] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7308.216358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7316.410791] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7316.730317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7324.130916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.135036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.149871] Lustre: Skipped 5 previous similar messages [ 7324.158164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7324.207529] Lustre: Skipped 3 previous similar messages [ 7326.901463] Lustre: server umount lustre-OST0000 complete [ 7346.144153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7346.639082] Lustre: server umount lustre-MDT0000 complete [ 7349.729713] LustreError: 212499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.784436] LustreError: 212499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7350.691487] LustreError: 211680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483693 with bad export cookie 726391629350733810 [ 7350.709077] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.711475] LustreError: 211680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7351.080300] Lustre: server umount lustre-MDT0001 complete [ 7355.443142] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:14:56 (1787483696) [ 7373.344499] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7384.183962] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7417.161988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.340947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7417.384494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7417.464274] Lustre: lustre-MDT0000: new disk, initializing [ 7417.584374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.587017] Lustre: Skipped 2 previous similar messages [ 7417.621819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7421.599735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7430.010808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.072673] Lustre: 218271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7430.274920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7434.752179] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7443.134924] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7448.186233] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7454.179496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.408403] Lustre: 220238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7456.008836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7459.791886] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7468.544924] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7473.110354] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7473.305199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7477.637929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7477.805443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7481.824990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7481.828513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.838561] LustreError: Skipped 1 previous similar message [ 7481.839941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.839947] Lustre: Skipped 6 previous similar messages [ 7481.875239] Lustre: Skipped 4 previous similar messages [ 7485.863338] Lustre: server umount lustre-OST0000 complete [ 7495.560867] Lustre: server umount lustre-MDT0000 complete [ 7497.195168] LustreError: 218280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.209907] LustreError: 218280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7498.821693] LustreError: 220237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483841 with bad export cookie 726391629350734692 [ 7498.822610] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7498.833695] LustreError: 220237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7499.246930] Lustre: server umount lustre-MDT0001 complete [ 7506.075133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.671564] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7517.067091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.579476] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7527.221603] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7532.108449] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7538.451570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.193947] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7550.362112] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.704937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.711342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7561.721623] Lustre: Skipped 2 previous similar messages [ 7561.722588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7561.722594] Lustre: Skipped 4 previous similar messages [ 7561.764248] LustreError: Skipped 1 previous similar message [ 7565.765566] Lustre: server umount lustre-OST0000 complete [ 7575.408540] Lustre: server umount lustre-MDT0000 complete [ 7579.161319] LustreError: 222435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483922 with bad export cookie 726391629350735574 [ 7579.170348] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.177085] LustreError: 222435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7579.441470] Lustre: server umount lustre-MDT0001 complete [ 7688.426286] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7689.930741] Key type lgssc unregistered [ 7690.092907] LNet: 226119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7690.097710] LNetError: 226119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7690.106607] LNet: Removed LNI 192.168.202.132@tcp [ 7690.544225] Key type .llcrypt unregistered [ 7690.546880] Key type ._llcrypt unregistered [ 7703.357502] Key type ._llcrypt registered [ 7703.359709] Key type .llcrypt registered [ 7703.416903] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7712.692917] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7713.557403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7713.592566] alg: No test for adler32 (adler32-zlib) [ 7714.585705] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 7714.715218] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7716.343190] Key type lgssc registered [ 7716.958248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7735.190423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7735.221798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.396563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7736.408510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7736.455240] Lustre: lustre-MDT0000: new disk, initializing [ 7736.496544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.505545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7739.221992] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7745.443387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.521632] Lustre: 229464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7745.563094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7745.565217] Lustre: Skipped 1 previous similar message [ 7745.610830] Lustre: lustre-MDT0001: new disk, initializing [ 7745.672258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7745.705519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7745.712533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7748.229217] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7753.694797] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7756.653632] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7760.298713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.443982] Lustre: lustre-OST0000: new disk, initializing [ 7760.446139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7760.449378] Lustre: 231433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7760.496283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.292340] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7766.031553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7766.037397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7766.074586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7769.515527] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7771.593399] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7771.701172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7773.989201] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7774.135529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7776.225591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.226536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.228642] LustreError: Skipped 1 previous similar message [ 7776.235247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.239504] Lustre: Skipped 1 previous similar message [ 7781.161064] Lustre: server umount lustre-OST0000 complete [ 7786.465527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.473830] Lustre: Skipped 1 previous similar message [ 7786.477461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.482240] Lustre: Skipped 1 previous similar message [ 7791.590173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.594905] Lustre: Skipped 1 previous similar message [ 7796.706342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.708306] Lustre: Skipped 1 previous similar message [ 7797.215432] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7797.300416] Lustre: server umount lustre-MDT0000 complete [ 7799.208362] LustreError: 229456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484142 with bad export cookie 1716866099102609921 [ 7799.212130] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.217136] LustreError: 229456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7799.420455] Lustre: server umount lustre-MDT0001 complete [ 7808.763715] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7813.395092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.605076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.329967] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7818.723866] LustreError: 234362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.730373] LustreError: 234362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7819.969513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.180072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.304539] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7823.954236] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7827.481390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.118210] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7834.335448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7834.430787] Lustre: lustre-OST0001: new disk, initializing [ 7834.432965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7834.435793] Lustre: 236669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.493675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7834.497100] Lustre: Skipped 1 previous similar message [ 7835.029909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7835.035519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7835.096467] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7837.422659] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7845.180165] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7849.833615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7855.948536] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:23:18 (1787484198) [ 7858.659868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.661985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.664780] Lustre: Skipped 3 previous similar messages [ 7858.668799] Lustre: Skipped 3 previous similar messages [ 7859.047972] Lustre: server umount lustre-MDT0000 complete [ 7861.018627] LustreError: 234343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484204 with bad export cookie 1716866099102610810 [ 7861.021468] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7861.023990] LustreError: 234343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7861.193826] Lustre: server umount lustre-MDT0001 complete [ 7872.987340] Lustre: server umount lustre-OST0000 complete [ 7884.164496] Lustre: server umount lustre-OST0001 complete [ 7887.499528] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7892.590586] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7910.643161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7910.792665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7910.808434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7910.854947] Lustre: lustre-MDT0000: new disk, initializing [ 7910.901825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.911695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7912.986813] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7918.149531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.205913] Lustre: 240609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7918.338444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7920.356331] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7925.822869] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7928.756413] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7932.602407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.761612] Lustre: lustre-OST0000: new disk, initializing [ 7932.764644] Lustre: Skipped 1 previous similar message [ 7932.767704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7932.770943] Lustre: Skipped 2 previous similar messages [ 7932.774958] Lustre: 242575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7934.617350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7934.622638] Lustre: Skipped 1 previous similar message [ 7934.625988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7934.693195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7936.447889] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7942.069398] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7944.113396] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7944.222617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7946.559797] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7946.680892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7950.309891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7950.310394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7950.314103] LustreError: Skipped 1 previous similar message [ 7950.318896] Lustre: Skipped 1 previous similar message [ 7950.319725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.324751] Lustre: Skipped 1 previous similar message [ 7953.772953] Lustre: server umount lustre-OST0000 complete [ 7959.520420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7959.524398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.529353] Lustre: Skipped 1 previous similar message [ 7961.778198] Lustre: server umount lustre-MDT0000 complete [ 7963.383134] LustreError: 240603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484306 with bad export cookie 1716866099102612896 [ 7963.384370] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.389805] LustreError: 240603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7963.542354] Lustre: server umount lustre-MDT0001 complete [ 7967.331262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.525659] LustreError: 244789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7967.557237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.560013] Lustre: Skipped 2 previous similar messages [ 7969.290809] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7972.832485] LustreError: 244790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.280841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.271319] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7978.701989] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7981.492666] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7984.838854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.753698] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7991.338236] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8135.650237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.654632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.661762] Lustre: Skipped 1 previous similar message [ 8135.665209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.667597] Lustre: Skipped 3 previous similar messages [ 8141.407491] Lustre: server umount lustre-OST0000 complete [ 8149.369099] Lustre: server umount lustre-MDT0000 complete [ 8151.032154] LustreError: 244771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484494 with bad export cookie 1716866099102613785 [ 8151.036201] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.036743] LustreError: 244771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8151.162581] Lustre: server umount lustre-MDT0001 complete [ 8257.203488] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8258.436963] Key type lgssc unregistered [ 8258.585670] LNet: 248395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8258.589488] LNetError: 248395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8258.602362] LNet: Removed LNI 192.168.202.132@tcp [ 8259.007482] Key type .llcrypt unregistered [ 8259.008761] Key type ._llcrypt unregistered [ 8269.286679] Key type ._llcrypt registered [ 8269.288166] Key type .llcrypt registered [ 8269.336889] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8276.786483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8277.302993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8277.323669] alg: No test for adler32 (adler32-zlib) [ 8278.213954] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 8278.360811] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8279.967770] Key type lgssc registered [ 8280.424472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8297.798219] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8301.940631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8301.951684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.049470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8303.064604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8303.102235] Lustre: lustre-MDT0000: new disk, initializing [ 8303.132171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.139750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8304.769076] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8309.570309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.620076] Lustre: 252434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8309.644525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8309.646549] Lustre: Skipped 1 previous similar message [ 8309.683872] Lustre: lustre-MDT0001: new disk, initializing [ 8309.724261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.738046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8309.742869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8311.385057] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8313.878417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8316.295968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8316.409079] Lustre: lustre-OST0000: new disk, initializing [ 8316.411446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8316.415293] Lustre: 254067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8316.452423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.576118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8316.580451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8316.626871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8318.636278] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8323.332089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8323.411280] Lustre: lustre-OST0001: new disk, initializing [ 8323.414306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8323.418973] Lustre: 254938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8323.472367] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8325.061672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8325.065263] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8325.099070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8325.728484] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8332.859303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8335.862300] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8339.913195] Lustre: server umount lustre-MDT0000 complete [ 8340.449954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.450066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.458560] Lustre: Skipped 2 previous similar messages [ 8341.394613] LustreError: 252428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484684 with bad export cookie 14313065812949834601 [ 8341.396930] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8341.399629] LustreError: 252428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8345.568882] LustreError: 252443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8345.570168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.572609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8345.578518] LustreError: 252443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8345.590535] Lustre: Skipped 2 previous similar messages [ 8346.592734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8346.592757] LustreError: 252442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8346.595911] Lustre: Skipped 2 previous similar messages [ 8346.602048] LustreError: 252442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8347.590947] Lustre: server umount lustre-MDT0001 complete [ 8349.188435] Lustre: server umount lustre-OST0000 complete [ 8350.803204] Lustre: server umount lustre-OST0001 complete [ 8356.723643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8360.990251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8364.661087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8368.665982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8376.387423] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8380.001308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.009306] Lustre: lustre-MDT0000: reset Object Index mappings [ 8380.153391] LustreError: 259938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.171419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.642238] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8384.936871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8384.947732] Lustre: lustre-MDT0001: reset Object Index mappings [ 8387.106330] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8388.283782] Lustre: 261080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8390.876151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8391.044679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.048027] Lustre: Skipped 1 previous similar message [ 8393.298123] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8395.170751] LustreError: 261434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.366255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8398.677877] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8404.970440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8404.970729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8412.883503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8414.372539] Lustre: 262970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8420.324250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.326554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8420.329151] Lustre: Skipped 2 previous similar messages [ 8420.333623] Lustre: Skipped 3 previous similar messages [ 8422.126930] Lustre: server umount lustre-MDT0000 complete [ 8423.503776] LustreError: 259920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484766 with bad export cookie 14313065812949841377 [ 8423.507149] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.508094] LustreError: 259920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8423.642123] Lustre: server umount lustre-MDT0001 complete [ 8435.666228] Lustre: server umount lustre-OST0000 complete [ 8447.552186] Lustre: server umount lustre-OST0001 complete [ 8457.220921] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:33:19 (1787484799) [ 8465.410989] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8470.089945] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8488.355287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.479138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8488.497521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8488.542404] Lustre: lustre-MDT0000: new disk, initializing [ 8488.574232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.577351] Lustre: Skipped 1 previous similar message [ 8488.585752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8490.750993] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8496.446728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.516611] Lustre: 267219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8496.552175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8496.554229] Lustre: Skipped 1 previous similar message [ 8496.592611] Lustre: lustre-MDT0001: new disk, initializing [ 8496.645566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8496.649400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8498.522507] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8503.457296] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8506.003886] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8509.275563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.407904] Lustre: 269186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8511.203708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8511.270329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8512.666557] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8517.355834] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8519.446662] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8519.546782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8521.422445] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8521.562743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8526.818619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.819145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.822872] Lustre: Skipped 1 previous similar message [ 8526.825921] LustreError: Skipped 1 previous similar message [ 8526.828398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8526.833527] Lustre: Skipped 1 previous similar message [ 8528.604437] Lustre: server umount lustre-OST0000 complete [ 8531.938266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.956676] Lustre: Skipped 1 previous similar message [ 8531.961623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8531.965380] Lustre: Skipped 1 previous similar message [ 8536.563927] Lustre: server umount lustre-MDT0000 complete [ 8537.056463] LustreError: 267229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.064628] LustreError: 267229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8538.303896] LustreError: 268012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484881 with bad export cookie 14313065812949843449 [ 8538.304221] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8538.307273] LustreError: 268012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8538.480384] Lustre: server umount lustre-MDT0001 complete [ 8547.608184] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8552.929373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.154830] LustreError: 272114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.188233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.192136] Lustre: Skipped 2 previous similar messages [ 8555.381728] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8559.908460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.055964] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8563.496559] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8566.947732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.282172] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8574.377920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8574.463391] Lustre: lustre-OST0001: new disk, initializing [ 8574.465422] Lustre: Skipped 1 previous similar message [ 8574.467972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8574.470218] Lustre: Skipped 1 previous similar message [ 8574.472990] Lustre: 274421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8576.277036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8576.281099] Lustre: Skipped 1 previous similar message [ 8576.283613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8576.346486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8577.440588] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8583.215936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8584.945985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8587.485920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8587.562890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8590.816528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8590.820147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.825412] Lustre: Skipped 1 previous similar message [ 8590.827915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.351382] Lustre: server umount lustre-MDT0000 complete [ 8598.181328] LustreError: 274420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787484941 with bad export cookie 14313065812949844324 [ 8598.184935] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8598.187366] LustreError: 274420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8598.348845] Lustre: server umount lustre-MDT0001 complete [ 8610.249714] Lustre: server umount lustre-OST0000 complete [ 8620.971283] Lustre: server umount lustre-OST0001 complete [ 8628.047796] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8632.380991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.563175] LustreError: 277474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.573645] LustreError: 277474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8632.593736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.599022] Lustre: Skipped 3 previous similar messages [ 8634.321973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8637.850964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.716578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8640.832233] Lustre: 278614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8643.358627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8646.603217] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8650.878035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8653.690850] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8657.388885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8657.395960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8662.509416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8664.035729] Lustre: 280493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8666.108361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8666.160869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8669.972565] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:36:52 (1787485012) [ 8682.981446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.982635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.986796] Lustre: Skipped 5 previous similar messages [ 8682.989423] Lustre: Skipped 5 previous similar messages [ 8687.971611] Lustre: server umount lustre-MDT0000 complete [ 8688.098329] LustreError: 277475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8688.104104] LustreError: 277475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8689.436615] LustreError: 280495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485032 with bad export cookie 14313065812949850645 [ 8689.439805] LustreError: 280495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8689.441043] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8695.623223] Lustre: server umount lustre-MDT0001 complete [ 8703.366513] Lustre: server umount lustre-OST0000 complete [ 8719.327274] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8719.416312] Lustre: server umount lustre-OST0001 complete [ 8722.317432] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8726.746266] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8745.738892] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8750.337796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.450753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8750.471122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8750.516426] Lustre: lustre-MDT0000: new disk, initializing [ 8750.568160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8752.304810] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8757.398507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.448673] Lustre: 284723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8757.584890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.204960] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8764.455716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.570027] Lustre: 286357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8764.603640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.605686] Lustre: Skipped 5 previous similar messages [ 8766.422083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8767.039022] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8772.340772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8774.195055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8774.723207] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8779.462081] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8785.746856] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:38:48 (1787485128) [ 8788.448327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8788.451774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.457102] Lustre: Skipped 3 previous similar messages [ 8788.459720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8788.462786] Lustre: Skipped 4 previous similar messages [ 8792.740424] Lustre: server umount lustre-MDT0000 complete [ 8794.104458] LustreError: 286356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485137 with bad export cookie 14313065812949858520 [ 8794.109105] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8794.109766] LustreError: 286356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8794.236281] Lustre: server umount lustre-MDT0001 complete [ 8804.944920] Lustre: server umount lustre-OST0000 complete [ 8816.678064] Lustre: server umount lustre-OST0001 complete [ 8819.999055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.187160] LustreError: 289473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.194456] LustreError: 289473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8821.789753] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8825.082032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.947949] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8829.872841] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8832.064733] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8834.621918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8836.910239] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8839.570394] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8856.700922] loop1: detected capacity change from 0 to 204800000 [ 8856.788736] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8856.792943] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8856.797968] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8858.640503] loop1: detected capacity change from 0 to 204800000 [ 8860.676829] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.709777] LustreError: 289462:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12945: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8860.717545] LustreError: 289462:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12945 [ 8860.720907] LustreError: 289462:0:(mgs_handler.c:517:mgs_target_reg()) fs12945-MDT0001: Error checking target index: rc = -13 [ 8860.725235] LustreError: fs12945-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8860.729579] LustreError: 293210:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8860.733363] LustreError: 293210:0:(tgt_mount.c:2129:server_put_super()) no obd fs12945-MDT0001 [ 8860.736400] LustreError: 293210:0:(tgt_mount.c:116:server_deregister_mount()) fs12945-MDT0001 not registered [ 8860.748520] Lustre: server umount fs12945-MDT0001 complete [ 8860.750783] LustreError: 293210:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8863.757650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.787916] Lustre: Setting parameter fs12945-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12945-MDT0001 [ 8863.791870] Lustre: Skipped 2 previous similar messages [ 8863.808654] Lustre: srv-fs12945-MDT0001: No data found on store. Initialize space. [ 8863.810663] Lustre: Skipped 4 previous similar messages [ 8863.846345] Lustre: fs12945-MDT0001: new disk, initializing [ 8863.848013] Lustre: Skipped 3 previous similar messages [ 8863.889495] Lustre: srv-fs12945-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8863.892682] Lustre: 293678:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12945-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8863.896145] LustreError: 293678:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12945-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8865.370957] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8868.351661] Lustre: server umount fs12945-MDT0001 complete [ 8872.884364] loop1: detected capacity change from 0 to 204800000 [ 8874.900525] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.017731] Lustre: 295241:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12945-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8875.022109] LustreError: 295241:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12945-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8876.384281] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8878.223670] Lustre: server umount fs12945-MDT0001 complete [ 8886.201338] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 8886.204836] Lustre: Skipped 4 previous similar messages [ 8886.242256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.246752] Lustre: Skipped 5 previous similar messages [ 8887.902218] Lustre: server umount lustre-MDT0000 complete [ 8891.206706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.296611] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.024460] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8896.012724] loop1: detected capacity change from 0 to 204800000 [ 8896.484982] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8898.109574] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.254077] Lustre: 297685:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12945-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8898.257929] LustreError: 297685:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12945-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8899.915110] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8901.964546] Lustre: server umount fs12945-MDT0001 complete [ 8911.841374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8915.221419] Lustre: server umount lustre-OST0000 complete [ 8931.295135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8931.354673] Lustre: server umount lustre-MDT0000 complete [ 8932.533706] LustreError: 289454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485275 with bad export cookie 14313065812949861194 [ 8932.534300] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8932.537344] LustreError: 289454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8932.642292] Lustre: server umount lustre-MDT0001 complete [ 8936.273503] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8937.394339] Key type lgssc unregistered [ 8937.529357] LNet: 300028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8937.532625] LNetError: 300028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8937.539375] LNet: Removed LNI 192.168.202.132@tcp [ 8937.883324] Key type .llcrypt unregistered [ 8937.884426] Key type ._llcrypt unregistered [ 8946.417291] Key type ._llcrypt registered [ 8946.418278] Key type .llcrypt registered [ 8946.461394] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8946.837545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8946.902348] alg: No test for adler32 (adler32-zlib) [ 8947.819464] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 8947.945988] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8949.567188] Key type lgssc registered [ 8950.015571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8953.549318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8953.555525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.707903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.093451] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8959.075939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.238252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8960.692451] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8963.191354] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8965.046929] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8967.587341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8967.750633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.826648] LustreError: 303106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8969.834839] LustreError: 303106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8969.940585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8972.694674] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8975.333834] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.425361] LustreError: 303105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.432852] LustreError: 303105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8983.996547] LustreError: 303780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.103519] LustreError: 303107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.108368] LustreError: 303107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8990.189170] loop1: detected capacity change from 0 to 204800000 [ 8990.262930] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8990.268059] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8990.275412] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8992.052257] loop1: detected capacity change from 0 to 204800000 [ 8993.930117] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8993.978663] Lustre: fs22360-OST0000: new disk, initializing [ 8993.980706] Lustre: srv-fs22360-OST0000: No data found on store. Initialize space. [ 8993.983428] Lustre: 305032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8994.007604] Lustre: fs22360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8996.000432] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8999.343622] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.350290] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8999.454391] Lustre: server umount fs22360-OST0000 complete [ 9003.611428] loop1: detected capacity change from 0 to 204800000 [ 9005.596433] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9005.669629] Lustre: fs22360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9007.743975] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9010.794187] loop2: detected capacity change from 0 to 204800000 [ 9010.876161] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9010.881635] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9010.886898] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9012.712403] loop2: detected capacity change from 0 to 204800000 [ 9014.617508] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9014.648718] LustreError: fs22360-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9014.654121] LustreError: fs22360-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9014.656810] LustreError: 308069:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9014.659426] LustreError: 308069:0:(tgt_mount.c:2129:server_put_super()) no obd fs22360-OST0001 [ 9014.661109] LustreError: 308069:0:(tgt_mount.c:116:server_deregister_mount()) fs22360-OST0001 not registered [ 9014.671241] Lustre: server umount fs22360-OST0001 complete [ 9014.672860] LustreError: 308069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9018.019327] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.064397] Lustre: fs22360-OST0001: new disk, initializing [ 9018.066700] Lustre: srv-fs22360-OST0001: No data found on store. Initialize space. [ 9018.070459] Lustre: 308618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9018.092662] Lustre: fs22360-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.823805] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.830522] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 9019.927868] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9023.411669] Lustre: server umount fs22360-OST0000 complete [ 9026.571204] Lustre: server umount fs22360-OST0001 complete [ 9030.618530] loop1: detected capacity change from 0 to 204800000 [ 9032.540846] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9034.546448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9036.881616] loop2: detected capacity change from 0 to 204800000 [ 9038.775029] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9038.848493] Lustre: fs22360-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9038.851696] Lustre: Skipped 1 previous similar message [ 9040.832851] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9043.842975] Lustre: server umount fs22360-OST0000 complete [ 9046.892310] Lustre: server umount fs22360-OST0001 complete [ 9051.106508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.107427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.109390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.112416] Lustre: Skipped 1 previous similar message [ 9051.115504] Lustre: Skipped 1 previous similar message [ 9056.224377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9056.224832] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9056.233240] LustreError: 303437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 9056.468336] Lustre: server umount lustre-OST0000 complete [ 9061.344642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.345602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.349382] Lustre: Skipped 1 previous similar message [ 9061.352080] Lustre: Skipped 2 previous similar messages [ 9066.464135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9066.466257] Lustre: Skipped 1 previous similar message [ 9071.584085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9071.586605] Lustre: Skipped 1 previous similar message [ 9072.095139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9072.164594] Lustre: server umount lustre-MDT0000 complete [ 9073.645432] LustreError: 301258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485416 with bad export cookie 1546977203378430506 [ 9073.646330] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9073.650390] LustreError: 301258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9073.776815] Lustre: server umount lustre-MDT0001 complete [ 9077.624688] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9078.715821] Key type lgssc unregistered [ 9078.846437] LNet: 314532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9078.850088] LNetError: 314532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9078.859485] LNet: Removed LNI 192.168.202.132@tcp [ 9079.187135] Key type .llcrypt unregistered [ 9079.188230] Key type ._llcrypt unregistered [ 9090.490162] Key type ._llcrypt registered [ 9090.491479] Key type .llcrypt registered [ 9090.544969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9096.953591] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9097.276246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9097.307087] alg: No test for adler32 (adler32-zlib) [ 9098.161038] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9098.244869] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9099.831169] Key type lgssc registered [ 9100.211901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9112.876285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9112.883075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.967589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9113.978727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9114.013080] Lustre: lustre-MDT0000: new disk, initializing [ 9114.037968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.044926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9115.392253] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9119.629345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.673919] Lustre: 318364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9119.695707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9119.697596] Lustre: Skipped 1 previous similar message [ 9119.731391] Lustre: lustre-MDT0001: new disk, initializing [ 9119.763318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.775873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9119.780113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9121.165554] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9124.983274] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9126.945262] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.245846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.345169] Lustre: lustre-OST0000: new disk, initializing [ 9129.347432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.350863] Lustre: 320329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9129.381794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.454140] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9135.489887] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9138.237616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9138.241763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9138.287892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9143.287478] Lustre: Mounted lustre-client - version 2.17.57_65_gc740cad [ 9144.066923] Lustre: Unmounted lustre-client [ 9153.508351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9153.509716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.511113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.528084] Lustre: Skipped 1 previous similar message [ 9157.546871] Lustre: server umount lustre-OST0000 complete [ 9160.673544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9160.680688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.691495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.695944] Lustre: Skipped 1 previous similar message [ 9163.746769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9163.749898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.314151] Lustre: server umount lustre-MDT0000 complete [ 9169.208833] LustreError: 319157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485512 with bad export cookie 15026988978534758648 [ 9169.215996] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9169.217405] LustreError: 319157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9169.480652] Lustre: server umount lustre-MDT0001 complete [ 9177.783940] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9180.591708] Key type lgssc unregistered [ 9191.135827] LNet: 322493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9191.141438] LNetError: 322493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9266.925682] LNet: Removed LNI 192.168.202.132@tcp [ 9267.509482] Key type .llcrypt unregistered [ 9267.511161] Key type ._llcrypt unregistered [ 9276.864693] Key type ._llcrypt registered [ 9276.866271] Key type .llcrypt registered [ 9276.911980] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9284.600645] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9285.022831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9285.079805] alg: No test for adler32 (adler32-zlib) [ 9285.959577] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9286.080466] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9287.679162] Key type lgssc registered [ 9288.242271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9302.805876] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:47:25 (1787485645) [ 9306.614337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9306.628351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.781338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9307.821749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9307.874199] Lustre: lustre-MDT0000: new disk, initializing [ 9307.917281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.926087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9310.293489] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9316.061378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.114803] Lustre: 325690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9316.147173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9316.151268] Lustre: Skipped 1 previous similar message [ 9316.200300] Lustre: lustre-MDT0001: new disk, initializing [ 9316.241435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9316.264279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9316.271216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9318.332502] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9323.634832] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9326.858512] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9330.602221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9330.736616] Lustre: lustre-OST0000: new disk, initializing [ 9330.738352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9330.741595] Lustre: 327656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9330.792640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.971811] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9338.421187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9338.430865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9338.501951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9339.228294] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9348.576331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9348.579608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.584594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.090385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9349.093762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.094109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.101618] Lustre: Skipped 1 previous similar message [ 9352.472930] Lustre: server umount lustre-OST0000 complete [ 9357.280966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9357.288800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9357.301644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.329914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9359.331294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.345399] Lustre: Skipped 1 previous similar message [ 9360.742777] Lustre: server umount lustre-MDT0000 complete [ 9362.924377] LustreError: 325682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485706 with bad export cookie 2150863922208681408 [ 9362.934022] LustreError: 325682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9362.940043] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9363.167195] Lustre: server umount lustre-MDT0001 complete [ 9370.442960] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9372.294418] Key type lgssc unregistered [ 9372.504748] LNet: 329564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9372.509725] LNetError: 329564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9372.520981] LNet: Removed LNI 192.168.202.132@tcp [ 9373.131198] Key type .llcrypt unregistered [ 9373.133207] Key type ._llcrypt unregistered [ 9386.560357] Key type ._llcrypt registered [ 9386.561953] Key type .llcrypt registered [ 9386.634453] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9387.333076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9387.348559] alg: No test for adler32 (adler32-zlib) [ 9388.315622] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9388.463382] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9390.079219] Key type lgssc registered [ 9390.793774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9395.431540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9395.445524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.724794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.150834] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9404.370780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.586499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9407.419386] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9412.260357] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9415.586799] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9419.650280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.826780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9423.753632] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9428.392708] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9435.400153] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9436.186245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9439.200506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.203341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.206500] Lustre: Skipped 1 previous similar message [ 9441.223569] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 9441.229121] Lustre: Skipped 2 previous similar messages [ 9444.321478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.324064] Lustre: Skipped 2 previous similar messages [ 9446.323281] LustreError: 332071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.338040] LustreError: 332071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9446.462479] Lustre: server umount lustre-MDT0000 complete [ 9448.637773] LustreError: 330797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485791 with bad export cookie 4158610490733883009 [ 9448.639920] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9448.649941] LustreError: 330797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9449.445127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.446740] LustreError: 333305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.456382] Lustre: Skipped 1 previous similar message [ 9449.459779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9451.440089] LustreError: 330811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.472564] LustreError: 333305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.472708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9453.492272] Lustre: Skipped 1 previous similar message [ 9454.907793] Lustre: server umount lustre-MDT0001 complete [ 9463.337143] Lustre: server umount lustre-OST0000 complete [ 9474.946715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.075610] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9483.771683] Lustre: 334997:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.132@tcp [ 9486.467664] Lustre: server umount MGS complete [ 9500.094465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.892342] LustreError: 335569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.009655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.678329] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9511.415223] LustreError: 335570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.455608] LustreError: 335570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9521.330047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.634469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9521.987368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9528.954369] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9537.172152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.415473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.716285] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9551.918555] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9553.342787] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9558.416772] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9560.008543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9566.183421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9566.190209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.201225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.233547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9568.241420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.180372] Lustre: server umount lustre-OST0000 complete [ 9578.471068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.473505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.493530] Lustre: Skipped 1 previous similar message [ 9579.736282] Lustre: server umount lustre-MDT0000 complete [ 9583.430619] LustreError: 335551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787485926 with bad export cookie 4158610490733883842 [ 9583.435513] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.439605] LustreError: 335551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9583.728071] Lustre: server umount lustre-MDT0001 complete [ 9594.045441] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9596.336804] Key type lgssc unregistered [ 9596.593878] LNet: 339007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9596.601334] LNetError: 339007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9596.627069] LNet: Removed LNI 192.168.202.132@tcp [ 9597.514546] Key type .llcrypt unregistered [ 9597.517154] Key type ._llcrypt unregistered [ 9615.018835] Key type ._llcrypt registered [ 9615.020949] Key type .llcrypt registered [ 9615.169635] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9630.647929] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9631.679109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.827773] alg: No test for adler32 (adler32-zlib) [ 9632.870631] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9633.040571] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9634.743217] Key type lgssc registered [ 9635.935459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9667.990450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9668.014594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9669.251963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9669.281775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9669.371640] Lustre: lustre-MDT0000: new disk, initializing [ 9669.434876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9669.451304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9673.246471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9681.820578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9681.930240] Lustre: 342168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9681.981141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9681.984128] Lustre: Skipped 1 previous similar message [ 9682.039418] Lustre: lustre-MDT0001: new disk, initializing [ 9682.153203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9682.173152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9682.190836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9685.812469] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9694.634610] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9699.889119] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9705.256450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9705.615610] Lustre: lustre-OST0000: new disk, initializing [ 9705.621644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9705.627877] Lustre: 344133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9705.721951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.443293] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9715.231433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9715.243094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9715.330852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9719.640914] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9723.526173] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9723.698418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9727.620762] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9727.787519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9730.534354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9730.541830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.556384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9731.046210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.064779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9731.071930] Lustre: Skipped 1 previous similar message [ 9735.619306] Lustre: server umount lustre-OST0000 complete [ 9738.728391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9738.736444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.750592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9741.281894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9741.283100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.708841] Lustre: server umount lustre-MDT0000 complete [ 9748.301438] LustreError: 342161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787486091 with bad export cookie 5865260775506602535 [ 9748.319226] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9748.631173] Lustre: server umount lustre-MDT0001 complete [ 9756.614726] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:54:58 (1787486098) [ 9771.755591] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9782.046531] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9813.557217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.852430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9813.878729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9813.918408] Lustre: lustre-MDT0000: new disk, initializing [ 9813.955346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9813.967879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9817.305922] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9826.597670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.680340] Lustre: 348892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9826.725316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9826.728186] Lustre: Skipped 1 previous similar message [ 9826.791447] Lustre: lustre-MDT0001: new disk, initializing [ 9826.940870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9826.999769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9827.012876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9831.319490] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9839.989930] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9844.646593] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9849.599898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.788613] Lustre: lustre-OST0000: new disk, initializing [ 9849.803427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9849.814949] Lustre: 350857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9849.900479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.863337] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9857.630655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9857.640320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9857.749754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9862.382880] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9888.741811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.753526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9888.755237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9888.766746] Lustre: Skipped 2 previous similar messages [ 9890.685586] Lustre: server umount lustre-OST0000 complete [ 9893.763621] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9898.978460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9898.982370] Lustre: Skipped 3 previous similar messages [ 9908.191148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9908.324529] Lustre: server umount lustre-MDT0000 complete [ 9911.095119] LustreError: 348883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787486254 with bad export cookie 5865260775506603424 [ 9911.099284] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.100432] LustreError: 348883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9911.297032] Lustre: server umount lustre-MDT0001 complete [ 9919.932336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9922.077602] Key type lgssc unregistered [ 9922.350426] LNet: 353012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9922.356853] LNetError: 353012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9923.436273] LNet: Removed LNI 192.168.202.132@tcp [ 9924.170920] Key type .llcrypt unregistered [ 9924.177569] Key type ._llcrypt unregistered [ 9988.629618] Key type ._llcrypt registered [ 9988.632321] Key type .llcrypt registered [ 9988.711147] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9988.927511] Key type .llcrypt unregistered [ 9988.932074] Key type ._llcrypt unregistered [10041.181131] Key type ._llcrypt registered [10041.182438] Key type .llcrypt registered [10041.264658] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10041.473948] Key type .llcrypt unregistered [10041.475775] Key type ._llcrypt unregistered [10060.799207] Key type ._llcrypt registered [10060.800340] Key type .llcrypt registered [10060.857518] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10061.047374] Key type .llcrypt unregistered [10061.048971] Key type ._llcrypt unregistered [10081.126565] Key type ._llcrypt registered [10081.128384] Key type .llcrypt registered [10081.185313] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10081.351629] Key type .llcrypt unregistered [10081.353392] Key type ._llcrypt unregistered [10124.678316] Key type ._llcrypt registered [10124.679595] Key type .llcrypt registered [10124.741664] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10124.966398] Key type .llcrypt unregistered [10124.968155] Key type ._llcrypt unregistered [10142.194502] Key type ._llcrypt registered [10142.195604] Key type .llcrypt registered [10142.248762] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10142.405454] Key type .llcrypt unregistered [10142.406759] Key type ._llcrypt unregistered [10159.396770] Key type ._llcrypt registered [10159.398456] Key type .llcrypt registered [10159.447849] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10159.608282] Key type .llcrypt unregistered [10159.610233] Key type ._llcrypt unregistered [10174.666019] Key type ._llcrypt registered [10174.667112] Key type .llcrypt registered [10174.726359] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10174.896614] Key type .llcrypt unregistered [10174.899137] Key type ._llcrypt unregistered [10192.459650] Key type ._llcrypt registered [10192.462240] Key type .llcrypt registered [10192.585288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10192.844880] Key type .llcrypt unregistered [10192.847128] Key type ._llcrypt unregistered [10211.282585] Key type ._llcrypt registered [10211.285414] Key type .llcrypt registered [10211.371485] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10211.573342] Key type .llcrypt unregistered [10211.574932] Key type ._llcrypt unregistered [10250.526558] Key type ._llcrypt registered [10250.528095] Key type .llcrypt registered [10250.582471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10250.783608] Key type .llcrypt unregistered [10250.784796] Key type ._llcrypt unregistered [10268.811574] Key type ._llcrypt registered [10268.813813] Key type .llcrypt registered [10268.885138] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10269.122475] Key type .llcrypt unregistered [10269.125810] Key type ._llcrypt unregistered [10285.495610] Key type ._llcrypt registered [10285.497051] Key type .llcrypt registered [10285.552273] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10285.743678] Key type .llcrypt unregistered [10285.745322] Key type ._llcrypt unregistered [10316.596394] Key type ._llcrypt registered [10316.603033] Key type .llcrypt registered [10316.680450] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10326.398056] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10326.973883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10327.072594] alg: No test for adler32 (adler32-zlib) [10328.050678] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [10328.186156] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10329.807184] Key type lgssc registered [10330.375689] Lustre: Echo OBD driver; http://www.lustre.org/ [10349.499800] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.506802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.621414] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10350.635178] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10350.686250] Lustre: tst250fs-MDT0000: new disk, initializing [10350.741615] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.750845] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10352.917339] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10359.509037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.614198] Lustre: 373011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10359.661458] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10359.665031] Lustre: Skipped 1 previous similar message [10359.717698] Lustre: tst250fs-MDT0001: new disk, initializing [10359.798975] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10359.825816] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10359.846441] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10362.600076] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10366.073552] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10370.512382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.727608] Lustre: tst250fs-OST0000: new disk, initializing [10370.730272] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10370.736378] Lustre: 374640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10370.814248] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10374.668804] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10379.813844] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10379.821868] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10379.905206] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10381.710118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10381.810562] Lustre: tst250fs-OST0001: new disk, initializing [10381.813700] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10381.818544] Lustre: 375513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10381.883992] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10384.864256] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10387.493834] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10387.499996] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10387.543931] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10388.470829] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10402.787973] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.793411] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10402.804281] Lustre: Skipped 3 previous similar messages [10402.805479] Lustre: Skipped 2 previous similar messages [10407.912263] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10407.915521] Lustre: Skipped 2 previous similar messages [10408.576120] Lustre: server umount tst250fs-MDT0000 complete [10410.378660] LustreError: 373001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787486753 with bad export cookie 4694240673041128119 [10410.383880] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10410.386709] LustreError: 373001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10410.549284] Lustre: server umount tst250fs-MDT0001 complete [10422.293513] Lustre: server umount tst250fs-OST0000 complete [10434.770627] Lustre: server umount tst250fs-OST0001 complete [10444.396869] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10448.558524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10464.717565] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:06:47 (1787486807) [10465.600919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10475.448613] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10480.748732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10496.708724] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10269 sec ========= 08:07:18 (1787486838) [10497.609911] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:07:19 (1787486839) === [10499.281452] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:07:21 (1787486841) === [10520.676125] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10522.496037] Key type lgssc unregistered [10522.716687] LNet: 384629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10522.721829] LNetError: 384629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10523.756081] LNet: Removed LNI 192.168.202.132@tcp [10524.256179] Key type .llcrypt unregistered [10524.258506] Key type ._llcrypt unregistered