[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 964654238 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002325] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021740] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026638] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029052] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031477] *** VALIDATE proc *** [ 0.032259] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042388] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.047913] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052285] rcu: Hierarchical SRCU implementation. [ 0.054896] smp: Bringing up secondary CPUs ... [ 0.056222] x86: Booting SMP configuration: [ 0.057364] .... node #0, CPUs: #1 #2 #3 [ 0.069143] smp: Brought up 1 node, 4 CPUs [ 0.071018] smpboot: Max logical packages: 1 [ 0.072075] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103068] node 0 deferred pages initialised in 29ms [ 0.106419] devtmpfs: initialized [ 0.107233] x86/mm: Memory block size: 128MB [ 0.110098] gcov: version magic: 0x41383552 [ 0.112381] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114491] pinctrl core: initialized pinctrl subsystem [ 0.115299] [ 0.116010] ************************************************************* [ 0.117160] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118015] ** ** [ 0.119010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120013] ** ** [ 0.121025] ** This means that this kernel is built to expose internal ** [ 0.122026] ** IOMMU data structures, which may compromise security on ** [ 0.123028] ** your system. ** [ 0.124033] ** ** [ 0.125028] ** If you see this message and you are not debugging the ** [ 0.126022] ** kernel, report this immediately to your vendor! ** [ 0.127021] ** ** [ 0.128022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129020] ************************************************************* [ 0.131000] NET: Registered protocol family 16 [ 0.133439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.134065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.135118] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.136890] cpuidle: using governor menu [ 0.137000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144652] PCI: Using configuration type 1 for base access [ 0.148179] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.165667] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.169117] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174278] cryptd: max_cpu_qlen set to 1000 [ 0.179050] ACPI: Added _OSI(Module Device) [ 0.182016] ACPI: Added _OSI(Processor Device) [ 0.184013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.193447] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201780] ACPI: Interpreter enabled [ 0.202082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203014] ACPI: Using IOAPIC for interrupt routing [ 0.204108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205469] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217600] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218106] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222553] acpiphp: Slot [2] registered [ 0.223127] acpiphp: Slot [5] registered [ 0.224169] acpiphp: Slot [6] registered [ 0.225120] acpiphp: Slot [7] registered [ 0.226171] acpiphp: Slot [8] registered [ 0.227096] acpiphp: Slot [9] registered [ 0.228148] acpiphp: Slot [10] registered [ 0.229110] acpiphp: Slot [3] registered [ 0.230141] acpiphp: Slot [4] registered [ 0.231272] acpiphp: Slot [11] registered [ 0.233143] acpiphp: Slot [12] registered [ 0.235112] acpiphp: Slot [13] registered [ 0.237114] acpiphp: Slot [14] registered [ 0.239109] acpiphp: Slot [15] registered [ 0.241174] acpiphp: Slot [16] registered [ 0.243116] acpiphp: Slot [17] registered [ 0.245216] acpiphp: Slot [18] registered [ 0.247237] acpiphp: Slot [19] registered [ 0.249148] acpiphp: Slot [20] registered [ 0.252130] acpiphp: Slot [21] registered [ 0.254242] acpiphp: Slot [22] registered [ 0.256112] acpiphp: Slot [23] registered [ 0.258186] acpiphp: Slot [24] registered [ 0.260097] acpiphp: Slot [25] registered [ 0.263122] acpiphp: Slot [26] registered [ 0.265127] acpiphp: Slot [27] registered [ 0.267282] acpiphp: Slot [28] registered [ 0.270128] acpiphp: Slot [29] registered [ 0.272137] acpiphp: Slot [30] registered [ 0.274174] acpiphp: Slot [31] registered [ 0.276097] PCI host bridge to bus 0000:00 [ 0.278106] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.282023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297354] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301351] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305626] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.324064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346664] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352085] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357060] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399390] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412039] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.554342] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569079] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.588017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.647051] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.660016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.674019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.701022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716638] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.729019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.743016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.771018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.781000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.788015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.798018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.836327] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.841540] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.844530] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.846276] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.851079] iommu: Default domain type: Passthrough [ 0.854000] SCSI subsystem initialized [ 0.854000] ACPI: bus type USB registered [ 0.855116] usbcore: registered new interface driver usbfs [ 0.857077] usbcore: registered new interface driver hub [ 0.859079] usbcore: registered new device driver usb [ 0.861280] pps_core: LinuxPPS API ver. 1 registered [ 0.863012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.867155] PTP clock support registered [ 0.869108] EDAC MC: Ver: 3.0.0 [ 0.870142] PCI: Using ACPI for IRQ routing [ 0.871000] NetLabel: Initializing [ 0.871008] NetLabel: domain hash size = 128 [ 0.873013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875083] NetLabel: unlabeled traffic allowed by default [ 0.881367] vgaarb: loaded [ 0.884312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.889019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.894903] clocksource: Switched to clocksource kvm-clock [ 1.153415] VFS: Disk quotas dquot_6.6.0 [ 1.161249] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.171156] *** VALIDATE ramfs *** [ 1.179156] *** VALIDATE hugetlbfs *** [ 1.180940] pnp: PnP ACPI init [ 1.191664] pnp: PnP ACPI: found 6 devices [ 1.248165] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.260372] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.263463] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.266771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.270208] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.273809] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.281409] NET: Registered protocol family 2 [ 1.291909] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.302594] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.308323] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.318850] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.323805] TCP: Hash tables configured (established 65536 bind 65536) [ 1.327545] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.332103] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.337735] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.341360] NET: Registered protocol family 1 [ 1.347065] RPC: Registered named UNIX socket transport module. [ 1.349766] RPC: Registered udp transport module. [ 1.354892] RPC: Registered tcp transport module. [ 1.356545] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.362832] NET: Registered protocol family 44 [ 1.364746] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.367419] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.369791] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.372911] PCI: CLS 0 bytes, default 64 [ 1.377612] Unpacking initramfs... [ 3.396518] debug: unmapping init [mem 0xffff93657cc54000-0xffff93657ffbffff] [ 3.402165] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.405577] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.409849] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.049093] Initialise system trusted keyrings [ 4.051272] Key type blacklist registered [ 4.053551] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.064141] zbud: loaded [ 4.067508] *** VALIDATE nfs *** [ 4.068776] *** VALIDATE nfs4 *** [ 4.070630] pstore: using deflate compression [ 4.075151] Platform Keyring initialized [ 4.217827] NET: Registered protocol family 38 [ 4.220797] Key type asymmetric registered [ 4.224601] Asymmetric key parser 'x509' registered [ 4.228156] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.234625] io scheduler mq-deadline registered [ 4.237543] io scheduler kyber registered [ 4.242595] io scheduler bfq registered [ 4.246116] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.250754] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.255650] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.261829] ACPI: Power Button [PWRF] [ 4.269023] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.277900] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.299758] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.313668] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.348893] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.386564] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.421057] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.426814] Non-volatile memory driver v1.3 [ 4.428833] Linux agpgart interface v0.103 [ 4.478107] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.486274] vda: detected capacity change from 0 to 74698752 [ 4.514852] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.523556] vdb: detected capacity change from 0 to 1073741824 [ 4.551869] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.557611] vdc: detected capacity change from 0 to 2621440000 [ 4.598352] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.603198] vdd: detected capacity change from 0 to 2621440000 [ 4.631177] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.635795] vde: detected capacity change from 0 to 4294967296 [ 4.668294] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.674857] vdf: detected capacity change from 0 to 4294967296 [ 4.688208] libphy: Fixed MDIO Bus: probed [ 4.704299] usbcore: registered new interface driver usbserial_generic [ 4.723803] usbserial: USB Serial support registered for generic [ 4.726299] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.730883] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.732969] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.735495] mousedev: PS/2 mouse device common for all mice [ 4.740312] rtc_cmos 00:05: RTC can wake from S4 [ 4.745274] rtc_cmos 00:05: registered as rtc0 [ 4.748782] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.755322] intel_pstate: CPU model not supported [ 4.761624] hid: raw HID events driver (C) Jiri Kosina [ 4.767722] usbcore: registered new interface driver usbhid [ 4.770460] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.773333] usbhid: USB HID core driver [ 4.773710] drop_monitor: Initializing network drop monitor service [ 4.773922] Initializing XFRM netlink socket [ 4.774724] NET: Registered protocol family 10 [ 4.783866] Segment Routing with IPv6 [ 4.794218] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.799463] NET: Registered protocol family 17 [ 4.800486] mpls_gso: MPLS GSO support [ 4.812461] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.830565] RAS: Correctable Errors collector initialized. [ 4.832894] AVX version of gcm_enc/dec engaged. [ 4.834948] AES CTR mode by8 optimization enabled [ 5.056860] sched_clock: Marking stable (5056841558, 0)->(7046685520, -1989843962) [ 5.065561] registered taskstats version 1 [ 5.069878] Loading compiled-in X.509 certificates [ 5.073302] zswap: loaded using pool lzo/zbud [ 5.154551] Key type big_key registered [ 5.199416] Key type encrypted registered [ 5.206928] ima: No TPM chip found, activating TPM-bypass! [ 5.212907] ima: Allocated hash algorithm: sha1 [ 5.218906] ima: No architecture policies found [ 5.223954] evm: Initialising EVM extended attributes: [ 5.231557] evm: security.selinux [ 5.235192] evm: security.ima [ 5.237447] evm: security.capability [ 5.244955] evm: HMAC attrs: 0x1 [ 5.253089] rtc_cmos 00:05: setting system clock to 2026-08-14 21:43:16 UTC (1786743796) [ 5.273695] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 5.286547] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 5.313126] Write protecting the kernel read-only data: 28672k [ 5.322876] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 5.331216] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 5.441894] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.474246] systemd[1]: Detected virtualization kvm. [ 5.475964] systemd[1]: Detected architecture x86-64. [ 5.477275] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.586097] systemd[1]: No hostname configured. [ 5.588043] systemd[1]: Set hostname to . [ 5.604299] random: systemd: uninitialized urandom read (16 bytes read) [ 5.606795] systemd[1]: Initializing machine ID from random generator. [ 5.867322] random: ln: uninitialized urandom read (6 bytes read) [ 6.209371] random: systemd: uninitialized urandom read (16 bytes read) [ 6.213488] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.224189] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.239295] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.924928] device-mapper: uevent: version 1.0.3 [ 7.927677] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.921298] virtio_net virtio0 ens2: renamed from eth0 [ 10.020648] random: fast init done [ 10.216024] scsi host0: ata_piix [ 10.339137] scsi host1: ata_piix [ 10.345135] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.360930] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.768497] random: crng init done [ 15.772341] random: 7 urandom warning(s) missed due to ratelimiting [ 17.763232] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.145808] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.726455] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.942808] SELinux: Disabled at runtime. [ 24.082294] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.105168] systemd[1]: Detected virtualization kvm. [ 24.107054] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.706289] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.709297] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.761914] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.772510] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.805509] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.847905] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.888504] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 27.351929] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 28.886041] hrtimer: interrupt took 17030951 ns [ OK ] Mounted /mnt. [ 29.100066] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.318897] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.414980] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.310247] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.473624] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.182615] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 38.597325] NFS: Registering the id_resolver key type [ 38.606945] Key type id_resolver registered [ 38.609328] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-server login: [ 108.863271] libcfs: loading out-of-tree module taints kernel. [ 108.958273] Key type ._llcrypt registered [ 108.960932] Key type .llcrypt registered [ 109.125756] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 128.397827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 129.872503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 129.882872] alg: No test for adler32 (adler32-zlib) [ 131.375934] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 132.374929] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 134.106282] Key type lgssc registered [ 136.825514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 156.194325] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.566991] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 209.073033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.164316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.434692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.531804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.657697] Lustre: lustre-MDT0000: new disk, initializing [ 210.772399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.832685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.933679] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.141313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.316363] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.414367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.432304] Lustre: Skipped 1 previous similar message [ 230.565047] Lustre: lustre-MDT0001: new disk, initializing [ 230.638628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.664405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.690479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.635214] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.391367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.904564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.126551] Lustre: lustre-OST0000: new disk, initializing [ 250.135210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.145197] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.208825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.704735] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.021809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.026428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.081054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.331918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.466715] Lustre: lustre-OST0001: new disk, initializing [ 269.468762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.472732] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.517380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.139424] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.072218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.087894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.190660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.992320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.673978] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.154381] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing check_logdir /tmp/testlogs/ [ 303.864558] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing yml_node [ 310.252541] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 314.762359] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 318.673567] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 320.714315] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 17:48:30 EDT 2026 [ 340.082367] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 342.324126] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.881800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 358.898643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.927324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.928303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.930949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.962015] Lustre: Skipped 3 previous similar messages [ 364.156829] Lustre: server umount lustre-MDT0000 complete [ 367.324673] LustreError: 7934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744158 with bad export cookie 1036073592645441994 [ 367.326654] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.333689] LustreError: 7934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 371.169203] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.170446] LustreError: 10109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.170650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.198359] Lustre: Skipped 3 previous similar messages [ 371.246092] LustreError: 10109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 372.194421] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.215534] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 373.745265] Lustre: server umount lustre-MDT0001 complete [ 377.467811] Lustre: server umount lustre-OST0000 complete [ 381.417626] Lustre: server umount lustre-OST0001 complete [ 387.322891] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 396.197329] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 429.141224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.479850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 429.569419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 429.667321] Lustre: lustre-MDT0000: new disk, initializing [ 429.750102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.778275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.734673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 444.363057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.542346] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.600164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.603863] Lustre: Skipped 1 previous similar message [ 444.668839] Lustre: lustre-MDT0001: new disk, initializing [ 444.759319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.805028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 444.819571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.516763] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 456.960551] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 463.031921] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.022599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.289306] Lustre: lustre-OST0000: new disk, initializing [ 469.293645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.303215] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.396281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.468854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.475041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.601530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.870918] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 482.440239] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.817460] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.987840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.545701] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.753980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.109135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.116507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.133640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.135586] Lustre: Skipped 1 previous similar message [ 496.146648] Lustre: Skipped 4 previous similar messages [ 498.561256] Lustre: server umount lustre-OST0000 complete [ 506.339263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.340883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.344136] LustreError: Skipped 1 previous similar message [ 506.344978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.387126] Lustre: Skipped 1 previous similar message [ 508.531846] Lustre: server umount lustre-MDT0000 complete [ 511.466221] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.495908] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 512.586739] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744303 with bad export cookie 1036073592645443527 [ 512.595121] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.603784] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 513.229623] Lustre: server umount lustre-MDT0001 complete [ 519.674380] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:51:49 (1786744309) [ 535.485567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.974146] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.042471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.578588] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 541.153178] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.180794] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 548.126584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.928978] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 559.416462] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.035816] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.442179] Lustre: Failing over lustre-MDT0000 [ 566.753890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.758520] LustreError: 19923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.770579] Lustre: Skipped 1 previous similar message [ 566.779128] LustreError: 19923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 566.881256] Lustre: server umount lustre-MDT0000 complete [ 583.144507] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.158917] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 586.479852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.581116] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.767129] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 591.845967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 591.855548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 591.887725] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 600.990537] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 607.203113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.207862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.225961] Lustre: Skipped 1 previous similar message [ 607.232290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.246404] Lustre: Skipped 2 previous similar messages [ 611.325640] Lustre: server umount lustre-MDT0000 complete [ 615.186342] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744406 with bad export cookie 1036073592645444941 [ 615.193294] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.196078] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 615.585206] Lustre: server umount lustre-MDT0001 complete [ 625.749953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.600925] LustreError: 23264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.631116] LustreError: 23264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 626.816751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.833253] Lustre: Skipped 2 previous similar messages [ 633.900725] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 642.588399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.947021] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 654.198286] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 660.933737] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 663.039091] Lustre: Failing over lustre-MDT0000 [ 663.283863] Lustre: server umount lustre-MDT0000 complete [ 663.532714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.535147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 681.796843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.937982] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.220639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 686.325669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 687.588379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 687.598754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 687.605660] Lustre: Skipped 1 previous similar message [ 687.629722] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 695.250325] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 702.947527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 702.949368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.964016] Lustre: Skipped 1 previous similar message [ 705.951939] Lustre: server umount lustre-MDT0000 complete [ 708.072247] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.088158] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 709.354548] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744500 with bad export cookie 1036073592645445879 [ 709.370715] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.371609] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 709.685955] Lustre: server umount lustre-MDT0001 complete [ 717.728914] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:55:08 (1786744508) [ 736.876715] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 746.811788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 786.901816] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 797.907117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.196266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 798.224298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 798.335523] Lustre: lustre-MDT0000: new disk, initializing [ 798.442199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.446872] Lustre: Skipped 2 previous similar messages [ 798.483602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 802.927427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 812.794322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.916957] Lustre: 29640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 813.084991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 817.794644] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 828.726267] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 835.740167] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 844.538749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 844.746757] Lustre: 31610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 846.390698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 846.461527] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 846.541815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 846.569825] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 852.055501] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 861.601209] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.363215] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 887.777483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.785793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.799056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.806079] Lustre: Skipped 4 previous similar messages [ 889.624940] Lustre: server umount lustre-OST0000 complete [ 909.281697] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 909.598427] Lustre: server umount lustre-MDT0000 complete [ 913.379953] LustreError: 29653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.398708] LustreError: 29653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 913.735785] LustreError: 30437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744704 with bad export cookie 1036073592645446285 [ 913.743150] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.752918] LustreError: 30437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 914.211947] Lustre: server umount lustre-MDT0001 complete [ 926.941732] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 931.277767] Key type lgssc unregistered [ 931.744987] LNet: 33592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.749723] LNetError: 33592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.774830] LNet: Removed LNI 192.168.202.154@tcp [ 932.895268] Key type .llcrypt unregistered [ 932.897247] Key type ._llcrypt unregistered [ 964.897413] Key type ._llcrypt registered [ 964.898628] Key type .llcrypt registered [ 964.992833] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 966.133328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.183959] alg: No test for adler32 (adler32-zlib) [ 967.267263] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 967.563882] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 969.303181] Key type lgssc registered [ 970.441957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 977.665436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 977.683250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.051344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.534230] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 989.665447] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.694320] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 992.165591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.589888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.593746] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1004.696489] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1011.245152] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1017.863807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.309387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.351792] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1024.486283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1032.160567] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1037.768978] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1051.113510] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:00:40 (1786744840) [ 1053.227943] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1055.440595] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1063.501825] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:00:53 (1786744853) [ 1072.912193] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:01:02 (1786744862) [ 1074.161550] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1083.553916] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1083.557719] Lustre: Skipped 2 previous similar messages [ 1085.251664] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:01:15 (1786744875) [ 1087.558538] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1087.562863] Lustre: Skipped 2 previous similar messages [ 1096.709290] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1096.715158] Lustre: Skipped 4 previous similar messages [ 1113.949662] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:01:43 (1786744903) [ 1117.157382] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1120.836711] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1130.559906] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:02:00 (1786744920) [ 1132.582550] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1132.588470] Lustre: Skipped 6 previous similar messages [ 1146.745474] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:02:16 (1786744936) [ 1162.728159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.745364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.788235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.793434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.796830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.831423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.884824] Lustre: server umount lustre-OST0000 complete [ 1176.036902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.042055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.050895] Lustre: Skipped 1 previous similar message [ 1176.069560] Lustre: Skipped 2 previous similar messages [ 1178.785632] Lustre: server umount lustre-MDT0000 complete [ 1181.153128] LustreError: 35735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.163585] LustreError: 35735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1183.048388] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786744974 with bad export cookie 4793259157915366854 [ 1183.051957] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.064225] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1183.403485] Lustre: server umount lustre-MDT0001 complete [ 1196.409421] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1200.482950] Key type lgssc unregistered [ 1201.059885] LNet: 42868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.070709] LNetError: 42868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.151301] LNet: Removed LNI 192.168.202.154@tcp [ 1203.918415] Key type .llcrypt unregistered [ 1203.925359] Key type ._llcrypt unregistered [ 1223.786240] Key type ._llcrypt registered [ 1223.788391] Key type .llcrypt registered [ 1223.894369] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 1241.000275] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1241.781618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.803104] alg: No test for adler32 (adler32-zlib) [ 1242.850327] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1243.174513] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1244.881804] Key type lgssc registered [ 1246.183046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1281.917115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1281.940203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.249354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1283.296880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1283.365625] Lustre: lustre-MDT0000: new disk, initializing [ 1283.547226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.563264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1287.152435] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1295.697867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.815875] Lustre: 46022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1295.871242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1295.874506] Lustre: Skipped 1 previous similar message [ 1295.928718] Lustre: lustre-MDT0001: new disk, initializing [ 1296.012818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.074173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1296.091336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1299.738439] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1307.691577] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1312.960521] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1319.390545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.650347] Lustre: lustre-OST0000: new disk, initializing [ 1319.652631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1319.656980] Lustre: 47990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1319.733633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.616380] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1329.773192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1329.794094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1329.910280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1334.607951] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1338.920825] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1339.132944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1343.991687] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1344.201863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1350.626593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1350.644622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.645869] LustreError: Skipped 1 previous similar message [ 1350.650854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.699717] Lustre: Skipped 1 previous similar message [ 1355.752085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.773072] Lustre: Skipped 2 previous similar messages [ 1360.352554] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.614770] Lustre: server umount lustre-OST0000 complete [ 1365.985448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.012467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.022343] Lustre: Skipped 1 previous similar message [ 1370.754653] Lustre: server umount lustre-MDT0000 complete [ 1371.105442] LustreError: 46033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.116307] LustreError: 46033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1374.315619] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745165 with bad export cookie 7809445939315812380 [ 1374.316930] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.328198] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1374.567567] Lustre: server umount lustre-MDT0001 complete [ 1383.306309] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:06:13 (1786745173) [ 1390.235328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.630436] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.737149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.567882] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1396.205152] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.293976] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.487909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.908339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.566541] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1413.696775] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1419.698787] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1426.231221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.558719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.517250] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1439.357524] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.381784] LustreError: 52980:0:(llog.c:451:llog_init_handle()) llog type is not specified! [ 1444.226505] Lustre: Modifying parameter general.timeout=1 in log params [ 1446.038793] Lustre: Modifying parameter general.timeout=3 in log params [ 1446.043397] Lustre: Skipped 1 previous similar message [ 1448.272727] Lustre: Modifying parameter general.timeout=5 in log params [ 1448.276383] Lustre: Skipped 1 previous similar message [ 1452.562882] Lustre: Modifying parameter general.timeout=9 in log params [ 1452.567490] Lustre: Skipped 3 previous similar messages [ 1461.169704] Lustre: Modifying parameter general.timeout=18 in log params [ 1461.175382] Lustre: Skipped 8 previous similar messages [ 1477.335074] Lustre: Modifying parameter general.timeout=33 in log params [ 1477.338634] Lustre: Skipped 14 previous similar messages [ 1509.817412] Lustre: Modifying parameter general.timeout=71 in log params [ 1509.820195] Lustre: Skipped 37 previous similar messages [ 1579.421981] Lustre: Disabling parameter general.timeout= in log params [ 1579.424497] Lustre: Skipped 58 previous similar messages [ 1583.141734] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:09:33 (1786745373) [ 1585.504955] Lustre: 59927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1587.929992] Lustre: 60038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1591.449307] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1591.460823] Lustre: 60236:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1621.145602] Lustre: 61803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1621.151344] Lustre: 61803:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1621.175437] LustreError: 61803:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1625.324801] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:10:15 (1786745415) [ 1638.371100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.378887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.391286] Lustre: Skipped 1 previous similar message [ 1638.398017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.073221] Lustre: server umount lustre-MDT0000 complete [ 1644.516472] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.540548] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1647.778946] LustreError: 61857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745439 with bad export cookie 7809445939315813262 [ 1647.785097] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.792043] LustreError: 61857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1648.060296] Lustre: server umount lustre-MDT0001 complete [ 1662.411766] Lustre: server umount lustre-OST0000 complete [ 1690.212952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.546047] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1690.769851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.614603] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1702.094517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.159516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1702.193266] Lustre: 64341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1706.642694] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1713.419500] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1719.133522] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.652190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.982215] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1726.332938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.338331] Lustre: Skipped 1 previous similar message [ 1733.228877] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1741.851840] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1743.341233] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1743.346709] Lustre: Skipped 23 previous similar messages [ 1743.388969] Lustre: 66912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1751.013225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.021381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.022337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.022342] Lustre: Skipped 3 previous similar messages [ 1751.035777] Lustre: Skipped 3 previous similar messages [ 1751.060255] LustreError: Skipped 1 previous similar message [ 1756.633912] Lustre: server umount lustre-OST0000 complete [ 1761.253937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.261754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.270620] Lustre: Skipped 1 previous similar message [ 1761.285029] Lustre: Skipped 4 previous similar messages [ 1766.639396] Lustre: server umount lustre-MDT0000 complete [ 1770.132126] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745561 with bad export cookie 7809445939315816475 [ 1770.135740] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.145303] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1770.487735] Lustre: server umount lustre-MDT0001 complete [ 1783.214307] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1786.596796] Key type lgssc unregistered [ 1787.003640] LNet: 68125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.012811] LNetError: 68125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.095780] LNet: Removed LNI 192.168.202.154@tcp [ 1789.416342] Key type .llcrypt unregistered [ 1789.419156] Key type ._llcrypt unregistered [ 1814.527996] Key type ._llcrypt registered [ 1814.533617] Key type .llcrypt registered [ 1814.707078] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1816.097856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.206228] alg: No test for adler32 (adler32-zlib) [ 1817.291275] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1817.583912] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1819.223258] Key type lgssc registered [ 1820.127632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.963796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.983829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.488135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.188644] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1838.055479] LustreError: 69539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.079078] LustreError: 69539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1839.662400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.290946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1845.064707] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1853.194997] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1859.370806] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.525246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1867.334559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.387864] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1881.447530] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1891.963495] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1900.519764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.528580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.530033] Lustre: Skipped 2 previous similar messages [ 1900.552448] Lustre: Skipped 2 previous similar messages [ 1905.650459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.663419] Lustre: Skipped 2 previous similar messages [ 1906.472872] Lustre: server umount lustre-MDT0000 complete [ 1909.360638] LustreError: 69519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745700 with bad export cookie 5608200966318968241 [ 1909.365575] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.367978] LustreError: 69519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1909.723409] Lustre: server umount lustre-MDT0001 complete [ 1923.785426] Lustre: server umount lustre-OST0000 complete [ 1947.314273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.582932] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1947.632351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.973943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.886789] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1962.758402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.832864] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1962.853708] Lustre: 74407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1963.517954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.896148] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1976.623054] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.761499] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.157541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.298462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1988.595482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.430290] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2001.874238] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2003.205264] Lustre: 76991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.466755] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2012.129884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.140545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.160740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.684045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.499435] Lustre: server umount lustre-OST0000 complete [ 2019.813290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.826579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.839109] Lustre: Skipped 1 previous similar message [ 2019.842848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.847024] Lustre: Skipped 1 previous similar message [ 2022.884633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.222329] Lustre: server umount lustre-MDT0000 complete [ 2028.002200] LustreError: 74418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.028974] LustreError: 74418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2028.957681] LustreError: 74400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786745820 with bad export cookie 5608200966318969081 [ 2028.964177] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.967407] LustreError: 74400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.467514] Lustre: server umount lustre-MDT0001 complete [ 2040.348131] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2043.235887] Key type lgssc unregistered [ 2043.406736] LNet: 78207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.411537] LNetError: 78207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.455058] LNet: Removed LNI 192.168.202.154@tcp [ 2045.395343] Key type .llcrypt unregistered [ 2045.397432] Key type ._llcrypt unregistered [ 2070.734826] Key type ._llcrypt registered [ 2070.736648] Key type .llcrypt registered [ 2070.833669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2072.009389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.179968] alg: No test for adler32 (adler32-zlib) [ 2073.593597] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2074.060919] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2075.791354] Key type lgssc registered [ 2077.276847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.327756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.369790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.949773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.565542] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2099.170069] LustreError: 79621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.189368] LustreError: 79621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2102.050332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.908893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.541847] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2114.700653] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2121.031115] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2127.691239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.158638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.741665] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2140.887442] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2150.269717] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2151.052342] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2153.168594] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2153.174745] Lustre: Skipped 1 previous similar message [ 2155.462948] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2155.473965] Lustre: Skipped 1 previous similar message [ 2160.221944] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2160.227663] Lustre: Skipped 4 previous similar messages [ 2168.419126] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2168.422719] Lustre: Skipped 9 previous similar messages [ 2184.733127] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2184.737524] Lustre: Skipped 21 previous similar messages [ 2217.128874] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2217.135282] Lustre: Skipped 34 previous similar messages [ 2242.968237] Lustre: 87717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2245.727037] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:20:35 (1786746035) [ 2247.421302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2249.182253] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:20:39 (1786746039) [ 2252.256463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.267817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.286394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2255.335690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2255.336062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.352353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.460672] Lustre: server umount lustre-OST0000 complete [ 2265.574849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.588415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.592671] Lustre: Skipped 1 previous similar message [ 2265.601440] Lustre: Skipped 1 previous similar message [ 2270.691792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.696843] Lustre: Skipped 2 previous similar messages [ 2274.790797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.812275] Lustre: Skipped 1 previous similar message [ 2276.319740] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2276.693175] Lustre: server umount lustre-MDT0000 complete [ 2279.912169] LustreError: 81780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.959893] LustreError: 81780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2282.160863] LustreError: 79601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746073 with bad export cookie 9392722485542373852 [ 2282.162468] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.174659] LustreError: 79601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2282.961706] Lustre: server umount lustre-MDT0001 complete [ 2295.337179] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2298.177783] Key type lgssc unregistered [ 2298.544543] LNet: 88968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.556772] LNetError: 88968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.581397] LNet: Removed LNI 192.168.202.154@tcp [ 2299.458515] Key type .llcrypt unregistered [ 2299.463651] Key type ._llcrypt unregistered [ 2305.102036] Key type ._llcrypt registered [ 2305.105296] Key type .llcrypt registered [ 2305.317279] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2309.959649] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules [ 2311.304707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.405561] alg: No test for adler32 (adler32-zlib) [ 2312.829696] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2313.175257] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2314.847392] Key type lgssc registered [ 2317.093548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2319.282436] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2339.974441] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2342.538648] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2350.499392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.510356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.895646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.810785] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2362.336890] LustreError: 91243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.346922] LustreError: 91243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2365.116417] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:22:35 (1786746155) [ 2367.459663] LustreError: 91244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.244290] LustreError: 91276:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2371.265662] LustreError: 91276:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2371.274873] LustreError: 91276:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2371.893333] Lustre: server umount lustre-MDT0000 complete [ 2388.412164] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2390.915055] Key type lgssc unregistered [ 2391.157656] LNet: 92771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.164521] LNetError: 92771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.196309] LNet: Removed LNI 192.168.202.154@tcp [ 2392.022950] Key type .llcrypt unregistered [ 2392.028480] Key type ._llcrypt unregistered [ 2409.441555] Key type ._llcrypt registered [ 2409.443658] Key type .llcrypt registered [ 2409.512207] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2410.542746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.665308] alg: No test for adler32 (adler32-zlib) [ 2411.838329] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2412.148775] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2413.807428] Key type lgssc registered [ 2415.067847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.224281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.256173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.832404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.602808] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2434.030558] LustreError: 93991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.075243] LustreError: 93991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2439.139422] LustreError: 93992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.654397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.455710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.642948] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2456.596043] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2462.040802] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2469.706043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.409692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.425849] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2483.859867] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2538.502337] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:25:27 (1786746327) [ 2540.239486] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2542.127733] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:25:32 (1786746332) [ 2545.120881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.139324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.143039] Lustre: Skipped 2 previous similar messages [ 2545.152081] Lustre: Skipped 2 previous similar messages [ 2550.244924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.249254] Lustre: Skipped 2 previous similar messages [ 2551.157991] Lustre: server umount lustre-MDT0000 complete [ 2555.363594] LustreError: 93991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.383856] LustreError: 93991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2555.388423] LustreError: 93972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746346 with bad export cookie 14868189450877970187 [ 2555.409880] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.421163] LustreError: 93972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2555.831190] Lustre: server umount lustre-MDT0001 complete [ 2569.156315] Lustre: server umount lustre-OST0000 complete [ 2580.059668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.475125] LustreError: 98286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.657399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.010442] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2586.081771] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.068807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.590277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.294679] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2605.013092] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2610.753633] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2622.921127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2623.190638] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2623.198797] LustreError: 98273:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2623.236035] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2623.252446] LustreError: 100370:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2623.261399] LustreError: 100370:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2623.268014] LustreError: 100370:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2623.360730] Lustre: server umount lustre-OST0000 complete [ 2623.362069] LustreError: 100370:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2632.455343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.613755] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2632.620176] LustreError: 98276:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2632.671565] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2632.683134] LustreError: 100837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2632.688926] LustreError: 100837:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2632.695098] LustreError: 100837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2632.785925] Lustre: server umount lustre-OST0000 complete [ 2632.789181] LustreError: 100837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2642.022273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.338721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2642.346060] Lustre: Found index 0 for lustre-OST0000, updating log [ 2642.352244] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2642.477264] Lustre: lustre-OST0000: new disk, initializing [ 2642.483844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2642.493936] Lustre: 101371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2642.858648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.984270] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2653.400794] Lustre: Failing over lustre-OST0000 [ 2653.549415] Lustre: server umount lustre-OST0000 complete [ 2659.808304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2659.817535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.829557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.420160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.426909] Lustre: Skipped 3 previous similar messages [ 2667.562152] Lustre: server umount lustre-MDT0000 complete [ 2669.537744] LustreError: 98305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.559223] LustreError: 98305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2671.144930] LustreError: 98268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746462 with bad export cookie 14868189450877972931 [ 2671.148141] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.152642] LustreError: 98268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2671.569342] Lustre: server umount lustre-MDT0001 complete [ 2680.399993] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:27:50 (1786746470) [ 2696.275755] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2706.433570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.008342] LustreError: 103715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.173124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.334947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2720.176569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.329505] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2732.293424] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2740.753121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.379475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.383994] Lustre: Skipped 1 previous similar message [ 2741.391585] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2745.464201] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2745.479177] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2745.538662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2748.782806] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2750.946318] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2750.963714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2752.013777] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2752.041669] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2752.046557] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2752.074834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2757.838636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.997429] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2758.095978] Lustre: lustre-OST0001: new disk, initializing [ 2758.101217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2758.108210] Lustre: 106037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2765.480865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2765.496127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2765.653601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2766.602256] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2773.971982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2781.737435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2786.284316] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.303286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2786.306770] Lustre: Skipped 2 previous similar messages [ 2786.326890] Lustre: Skipped 2 previous similar messages [ 2792.086826] Lustre: server umount lustre-MDT0001 complete [ 2794.884026] LustreError: 103711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.905261] LustreError: 103711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2802.229341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.374301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2802.384475] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2802.428911] Lustre: 103705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2803.697208] LustreError: 107579:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2803.704161] LustreError: 107579:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2803.709549] LustreError: 103707:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2803.799759] LustreError: 107590:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -17 [ 2803.815188] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2803.815188] [ 2807.282682] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2808.306391] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2817.641527] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:30:07 (1786746607) [ 2834.973289] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2850.105035] Lustre: 109649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2870.685343] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2874.384183] Lustre: 110784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2951.649937] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.654622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.668434] Lustre: Skipped 3 previous similar messages [ 2951.687701] Lustre: Skipped 6 previous similar messages [ 2956.282089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.303307] LustreError: Skipped 1 previous similar message [ 2956.314940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.868949] Lustre: server umount lustre-MDT0000 complete [ 2961.887636] LustreError: 103710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.906306] LustreError: 103710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2962.070358] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746753 with bad export cookie 14868189450877973932 [ 2962.072068] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2962.078345] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2962.353168] Lustre: server umount lustre-MDT0001 complete [ 2976.349300] Lustre: server umount lustre-OST0000 complete [ 2991.408427] Lustre: server umount lustre-OST0001 complete [ 2999.253222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3011.246589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3021.016688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3030.804233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3050.684635] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3062.332790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.354252] Lustre: lustre-MDT0000: reset Object Index mappings [ 3062.780126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.785536] Lustre: Skipped 2 previous similar messages [ 3066.074250] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3074.138457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.153473] Lustre: lustre-MDT0001: reset Object Index mappings [ 3079.583408] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3082.786485] Lustre: 115887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3090.901046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.426253] LustreError: 116240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.451976] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3099.468704] LustreError: 116240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3100.948809] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3106.837206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3110.768623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.580958] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3116.616541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3119.179127] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3128.382000] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3132.517285] Lustre: 117762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3178.005732] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:36:08 (1786746968) [ 3207.655997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3207.664850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.682206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3207.687538] Lustre: Skipped 4 previous similar messages [ 3210.743675] Lustre: server umount lustre-MDT0000 complete [ 3214.378575] LustreError: 114720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747005 with bad export cookie 14868189450878021007 [ 3214.379395] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3214.389951] LustreError: 114720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3221.129331] Lustre: server umount lustre-MDT0001 complete [ 3238.882349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.968659] Lustre: server umount lustre-OST0000 complete [ 3248.975017] Lustre: server umount lustre-OST0001 complete [ 3256.658871] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 3265.835382] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3306.148491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.753930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3306.831530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3307.102979] Lustre: lustre-MDT0000: new disk, initializing [ 3307.260548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.275904] Lustre: Skipped 3 previous similar messages [ 3307.308711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3311.444971] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3319.327966] Lustre: server umount lustre-MDT0000 complete [ 3330.434259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.705285] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3330.728679] Lustre: 122035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3330.755252] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3336.945619] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3345.729792] Lustre: server umount lustre-MDT0000 complete [ 3356.964408] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:39:06 (1786747146) [ 3358.495482] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3360.273124] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:39:10 (1786747150) [ 3382.056540] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3394.227191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.810447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3408.803250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.946168] Lustre: 124087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3409.049673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3409.059770] Lustre: Skipped 1 previous similar message [ 3409.212725] Lustre: lustre-MDT0001: new disk, initializing [ 3409.408241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3409.421235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3414.577165] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3420.972767] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3429.252481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.640673] Lustre: lustre-OST0000: new disk, initializing [ 3429.644081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3429.653352] Lustre: 125665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3431.164678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3431.172861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3431.280293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3435.371593] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3447.746434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.132776] Lustre: lustre-OST0001: new disk, initializing [ 3448.142541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3448.163680] Lustre: 126539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3449.728793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3449.747381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3449.949354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3457.667471] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3468.539762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3472.579792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3855.891489] Lustre: 132954:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3855.898569] Lustre: 132954:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3855.903453] Lustre: 132954:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3855.909721] Lustre: 132954:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3855.921115] Lustre: 132954:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3855.928624] Lustre: 132954:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4123.171805] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:51:53 (1786747913) [ 4141.027789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.030317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.041251] Lustre: Skipped 8 previous similar messages [ 4141.065158] Lustre: Skipped 11 previous similar messages [ 4146.147674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.158601] Lustre: Skipped 2 previous similar messages [ 4152.287382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4152.532720] Lustre: server umount lustre-MDT0000 complete [ 4155.947534] LustreError: 132966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747947 with bad export cookie 14868189450878086954 [ 4155.959574] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.962671] LustreError: 132966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4156.390365] LustreError: 127343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.391758] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.406368] LustreError: 127343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 4156.441162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4156.450991] Lustre: Skipped 5 previous similar messages [ 4162.575992] Lustre: server umount lustre-MDT0001 complete [ 4171.556162] Lustre: server umount lustre-OST0000 complete [ 4189.154142] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4189.469884] Lustre: server umount lustre-OST0001 complete [ 4195.509671] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4204.801443] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4239.367037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.649626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4239.691393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4239.815380] Lustre: lustre-MDT0000: new disk, initializing [ 4239.919955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.927222] Lustre: Skipped 5 previous similar messages [ 4239.946558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4244.613427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4255.263137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.337841] Lustre: 136005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4255.384034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4255.387525] Lustre: Skipped 1 previous similar message [ 4255.447581] Lustre: lustre-MDT0001: new disk, initializing [ 4255.598129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4255.613649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4260.503075] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4270.579398] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4276.574552] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.009161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.307358] Lustre: lustre-OST0000: new disk, initializing [ 4283.312139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4283.320186] Lustre: 137973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4283.437625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.442896] Lustre: Skipped 1 previous similar message [ 4284.925177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4284.932442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4285.053284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4289.967924] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4301.213173] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4312.540897] Lustre: lustre-MDD0000: changelog on [ 4316.680649] Lustre: lustre-MDD0001: changelog on [ 5914.230186] Lustre: lustre-MDD0001: changelog off [ 5916.702900] Lustre: 136010:0:(llog_cat.c:309:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5917.507410] Lustre: lustre-MDD0000: changelog off [ 5928.419734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.431405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.431476] Lustre: Skipped 2 previous similar messages [ 5928.434543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.470848] Lustre: Skipped 3 previous similar messages [ 5930.325708] Lustre: server umount lustre-OST0000 complete [ 5935.076443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5935.089949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.116782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.657086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5940.338645] Lustre: server umount lustre-MDT0000 complete [ 5943.785055] LustreError: 136011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.807995] LustreError: 136011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5944.808660] LustreError: 135996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749736 with bad export cookie 14868189450878906486 [ 5944.814815] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5944.833446] LustreError: 135996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5945.263728] Lustre: server umount lustre-MDT0001 complete [ 5960.235711] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5963.862536] Key type lgssc unregistered [ 5964.251560] LNet: 147081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5964.260977] LNetError: 147081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5964.289923] LNet: Removed LNI 192.168.202.154@tcp [ 5965.101155] Key type .llcrypt unregistered [ 5965.105880] Key type ._llcrypt unregistered [ 5986.060165] Key type ._llcrypt registered [ 5986.064033] Key type .llcrypt registered [ 5986.170447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6003.948609] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6005.226158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6005.314712] alg: No test for adler32 (adler32-zlib) [ 6006.429059] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6006.782765] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6008.499945] Key type lgssc registered [ 6010.026763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6046.285052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6046.305547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.520510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6047.549091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6047.619546] Lustre: lustre-MDT0000: new disk, initializing [ 6047.689383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.706481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6051.828926] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6062.538912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.634343] Lustre: 150285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6062.672446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6062.675729] Lustre: Skipped 1 previous similar message [ 6062.725685] Lustre: lustre-MDT0001: new disk, initializing [ 6062.824914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6062.889991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6062.897271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6067.770339] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6077.736411] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6084.748501] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6092.521909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.908919] Lustre: lustre-OST0000: new disk, initializing [ 6092.919499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6092.926126] Lustre: 152253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6093.031899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.802589] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6103.064381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6103.074957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6103.191538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6110.014242] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.966886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_module obdecho/obdecho [ 6121.111668] LustreError: 153258:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6121.131850] LustreError: 153258:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6124.132652] LustreError: 153354:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6144.489592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.494383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6144.508546] Lustre: Skipped 2 previous similar messages [ 6144.524273] Lustre: Skipped 2 previous similar messages [ 6147.319863] Lustre: server umount lustre-MDT0000 complete [ 6151.991132] LustreError: 150277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749943 with bad export cookie 15666134316391558730 [ 6152.017377] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6152.737158] Lustre: server umount lustre-MDT0001 complete [ 6168.379752] Lustre: server umount lustre-OST0000 complete [ 6189.965389] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6201.808619] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6240.207114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.623875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6240.652042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6240.809471] Lustre: lustre-MDT0000: new disk, initializing [ 6240.912031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.929437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6245.812973] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6256.168313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.308229] Lustre: 157074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6256.368933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6256.372846] Lustre: Skipped 1 previous similar message [ 6256.473279] Lustre: lustre-MDT0001: new disk, initializing [ 6256.557039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.597271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6256.618912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6260.943340] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6270.653626] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6277.069312] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.424460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.845212] Lustre: lustre-OST0000: new disk, initializing [ 6285.848428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6285.855160] Lustre: 159043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6285.929296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.056607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6293.067556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.241560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6293.455798] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6304.129741] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6308.819405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6309.206864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6314.945257] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6315.203410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6319.072373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.073943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.094050] LustreError: Skipped 1 previous similar message [ 6319.095439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.141788] Lustre: Skipped 1 previous similar message [ 6324.198180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.211326] Lustre: Skipped 2 previous similar messages [ 6329.313187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6329.333769] Lustre: Skipped 1 previous similar message [ 6331.872414] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6332.111666] Lustre: server umount lustre-OST0000 complete [ 6338.529134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6338.538577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.551301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.794180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.810380] Lustre: Skipped 5 previous similar messages [ 6350.303134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6350.482729] Lustre: server umount lustre-MDT0000 complete [ 6354.270931] LustreError: 157065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750145 with bad export cookie 15666134316391559689 [ 6354.275352] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6354.286846] LustreError: 157065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6354.695846] Lustre: server umount lustre-MDT0001 complete [ 6359.172396] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:29:09 (1786750149) [ 6361.410393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6363.594764] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:29:13 (1786750153) [ 6373.103500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.789936] LustreError: 161459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.817654] LustreError: 161459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6373.907230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6378.567708] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6378.976466] LustreError: 161460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.098538] LustreError: 161459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.675703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.414586] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6402.422051] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6408.324963] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6414.214478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6414.452268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6414.456759] Lustre: Skipped 1 previous similar message [ 6419.600225] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6428.717888] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6439.809954] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6441.424358] Lustre: server umount lustre-MDT0000 complete [ 6442.473782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.487198] Lustre: Skipped 1 previous similar message [ 6442.494726] LustreError: 163948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6442.523965] LustreError: 163948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6445.311851] LustreError: 161442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750236 with bad export cookie 15666134316391560571 [ 6445.313633] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.325082] LustreError: 161442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6447.587539] LustreError: 161455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.593584] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.607961] LustreError: 161455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6447.636131] Lustre: Skipped 2 previous similar messages [ 6451.732879] Lustre: server umount lustre-MDT0001 complete [ 6459.185403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6474.136936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.326109] LustreError: 165299:0:(llog.c:1658:llog_backup()) MGC192.168.202.154@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6474.339699] Lustre: 165299:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.154@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6475.232473] LustreError: 165303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.264724] LustreError: 165303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6495.715793] LustreError: 165303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.727691] LustreError: 165303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6530.925489] LustreError: 165305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.947066] LustreError: 165305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 6535.135319] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6535.138780] Lustre: 165299:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6576.118539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6581.174805] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6581.253761] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6589.829857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.074075] LustreError: 166043:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6591.081126] LustreError: 166043:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000001f465607 x1873542956319104/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786750382 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6591.099850] LustreError: 166043:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6597.112709] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6597.222511] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6604.171936] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6605.542559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6611.581557] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6613.636246] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6632.931071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6632.933385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.937630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.961500] Lustre: Skipped 1 previous similar message [ 6632.980444] Lustre: Skipped 5 previous similar messages [ 6645.216849] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6645.504806] Lustre: server umount lustre-MDT0000 complete [ 6648.293192] LustreError: 166869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6648.305382] LustreError: 166869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 6649.351192] LustreError: 161440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750440 with bad export cookie 15666134316391567025 [ 6649.366217] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6649.369597] LustreError: 161440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6649.834844] Lustre: server umount lustre-MDT0001 complete [ 6664.551089] Lustre: server umount lustre-OST0000 complete [ 6687.501526] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6697.127611] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6731.774386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.153978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6732.208443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6732.385657] Lustre: lustre-MDT0000: new disk, initializing [ 6732.503565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.510801] Lustre: Skipped 1 previous similar message [ 6732.527763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6737.038581] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6748.926794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.068245] Lustre: 170405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6749.407253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6753.887715] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6767.500482] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6774.325721] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6781.320798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6781.605642] Lustre: lustre-OST0000: new disk, initializing [ 6781.611939] Lustre: Skipped 1 previous similar message [ 6781.617876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6781.630140] Lustre: Skipped 2 previous similar messages [ 6781.642689] Lustre: 172375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6788.655356] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6791.277599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6791.287779] Lustre: Skipped 1 previous similar message [ 6791.297552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6791.442384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6799.017768] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.065383] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6803.534556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6808.648215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6808.957081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6812.141656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6812.143118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6812.147177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6812.147188] Lustre: Skipped 6 previous similar messages [ 6812.153809] Lustre: Skipped 2 previous similar messages [ 6812.177508] LustreError: Skipped 1 previous similar message [ 6817.509577] Lustre: server umount lustre-OST0000 complete [ 6827.995324] Lustre: server umount lustre-MDT0000 complete [ 6831.464407] LustreError: 170400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750622 with bad export cookie 15666134316391583860 [ 6831.478864] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6831.850398] Lustre: server umount lustre-MDT0001 complete [ 6840.949356] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:37:11 (1786750631) [ 6848.236406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.709494] LustreError: 174786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.099309] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6860.743244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.170961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6875.412880] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6881.886954] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6892.313181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.749994] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6906.078542] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6918.001491] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:38:27 (1786750707) [ 6921.696852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.702139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.716206] Lustre: Skipped 3 previous similar messages [ 6927.877954] Lustre: server umount lustre-OST0000 complete [ 6932.960436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6938.633466] Lustre: server umount lustre-MDT0000 complete [ 6943.923376] LustreError: 177562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750735 with bad export cookie 15666134316391584735 [ 6943.946401] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6943.949770] LustreError: 177562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6944.619155] Lustre: server umount lustre-MDT0001 complete [ 6957.061667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6959.374549] Key type lgssc unregistered [ 6959.599710] LNet: 178521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6959.607370] LNetError: 178521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6959.618783] LNet: Removed LNI 192.168.202.154@tcp [ 6960.418273] Key type .llcrypt unregistered [ 6960.420053] Key type ._llcrypt unregistered [ 6978.902963] Key type ._llcrypt registered [ 6978.905090] Key type .llcrypt registered [ 6979.010736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6979.968857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6980.008947] alg: No test for adler32 (adler32-zlib) [ 6981.006435] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 6981.225183] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6982.855138] Key type lgssc registered [ 6984.186258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6990.957929] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6990.969867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.559660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7011.679299] LustreError: 179762:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 7016.799483] LustreError: 179762:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7016.822757] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7016.836451] LustreError: MGC192.168.202.154@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7016.847683] LustreError: 179762:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7016.855043] LustreError: 179762:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7016.860039] LustreError: 179762:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7016.930199] Lustre: server umount lustre-OST0000 complete [ 7016.935134] LustreError: 179762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7025.896095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.400544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.973376] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7036.902323] LustreError: 180262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.927493] LustreError: 180262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7040.378489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.634113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.236815] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7052.298351] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7059.161730] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.670055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.900368] LustreError: 182076:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7065.908499] Lustre: 182076:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.154@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7066.010408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.795669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7078.424476] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7081.449917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7081.454258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.461428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.466715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.468434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.474812] Lustre: Skipped 1 previous similar message [ 7082.488608] Lustre: Skipped 2 previous similar messages [ 7085.538479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7085.542252] Lustre: Skipped 2 previous similar messages [ 7086.648374] Lustre: server umount lustre-MDT0000 complete [ 7090.343277] LustreError: 180242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750881 with bad export cookie 16367925527167627620 [ 7090.357713] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7090.361875] LustreError: 180242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7090.657136] LustreError: 180257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.658142] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.692076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7091.743382] LustreError: 180991:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004c72be78 x1873543977530112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7091.888353] Lustre: server umount lustre-MDT0001 complete [ 7095.992305] Lustre: server umount lustre-OST0000 complete [ 7105.321318] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7115.355549] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7149.729557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.963165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7149.997306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7150.125768] Lustre: lustre-MDT0000: new disk, initializing [ 7150.181586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.196746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7154.450836] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7164.470032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.607560] Lustre: 185598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7164.699909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7164.705746] Lustre: Skipped 1 previous similar message [ 7164.726366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7164.777427] Lustre: lustre-MDT0001: new disk, initializing [ 7164.826600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.846509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7164.856725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7169.051078] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7178.493584] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.036722] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.657829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.169734] Lustre: lustre-OST0000: new disk, initializing [ 7191.175093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7191.194335] Lustre: 187563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7191.370518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7199.550846] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7200.861127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7200.881609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7201.137404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7212.123509] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7218.202884] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7218.638953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7224.573342] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7224.859506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7226.848907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.850031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7226.859916] Lustre: Skipped 1 previous similar message [ 7226.876985] Lustre: Skipped 1 previous similar message [ 7233.030926] Lustre: server umount lustre-OST0000 complete [ 7241.696732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7241.703753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.059706] Lustre: server umount lustre-MDT0000 complete [ 7247.335230] LustreError: 185608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.358704] LustreError: 185608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7249.144949] LustreError: 186391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751040 with bad export cookie 16367925527167628341 [ 7249.151090] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.157033] LustreError: 186391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7249.422611] Lustre: server umount lustre-MDT0001 complete [ 7258.523430] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:44:08 (1786751048) [ 7275.312511] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7277.578633] Key type lgssc unregistered [ 7277.804178] LNet: 190515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.808226] LNetError: 190515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.820856] LNet: Removed LNI 192.168.202.154@tcp [ 7278.439911] Key type .llcrypt unregistered [ 7278.441745] Key type ._llcrypt unregistered [ 7296.695077] Key type ._llcrypt registered [ 7296.697447] Key type .llcrypt registered [ 7296.802915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7297.628771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.661773] alg: No test for adler32 (adler32-zlib) [ 7298.667796] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 7298.851645] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7300.495177] Key type lgssc registered [ 7301.770216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7310.175628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7310.217989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.007418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.521733] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7322.599156] LustreError: 191734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.636397] LustreError: 191734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7326.963559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.482141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7332.615850] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7341.099067] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7348.682557] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.397193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.778721] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7357.787090] LustreError: 193600:0:(llog.c:1699:llog_backup()) MGC192.168.202.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7357.800820] Lustre: 193600:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.154@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7358.009994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.794341] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7369.855396] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7371.568800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.582475] Lustre: Skipped 1 previous similar message [ 7371.583086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7371.602234] Lustre: Skipped 2 previous similar messages [ 7372.641967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.894983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.906850] Lustre: Skipped 4 previous similar messages [ 7377.969920] Lustre: server umount lustre-MDT0000 complete [ 7381.642767] LustreError: 191715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751172 with bad export cookie 15655038342160394447 [ 7381.663579] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7381.665896] LustreError: 191715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7382.943885] LustreError: 192466:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c2b6bd68 x1873544310974336/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7383.012974] LustreError: 191729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.035535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.051585] Lustre: Skipped 1 previous similar message [ 7383.155960] Lustre: server umount lustre-MDT0001 complete [ 7389.095222] Lustre: server umount lustre-OST0000 complete [ 7398.777993] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7407.641832] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7440.987327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.226654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7441.273728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7441.377836] Lustre: lustre-MDT0000: new disk, initializing [ 7441.429845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.453839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7445.087250] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7455.133310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.317458] Lustre: 197123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7455.394787] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7455.403149] Lustre: Skipped 1 previous similar message [ 7455.406971] LustreError: 197906:0:(llog.c:1699:llog_backup()) MGC192.168.202.154@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7455.424069] LustreError: 197906:0:(llog.c:1699:llog_backup()) Skipped 1 previous similar message [ 7455.434272] Lustre: 197906:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.154@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7455.450134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7455.454823] Lustre: Skipped 1 previous similar message [ 7455.554363] Lustre: lustre-MDT0001: new disk, initializing [ 7455.677835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.710148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7455.720701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7460.722220] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7471.123702] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7476.847626] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7483.952823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.242837] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7484.245541] Lustre: Skipped 1 previous similar message [ 7484.249755] LustreError: 199070:0:(llog.c:1699:llog_backup()) MGC192.168.202.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7484.258939] LustreError: 199070:0:(llog.c:1699:llog_backup()) Skipped 1 previous similar message [ 7484.262661] Lustre: 199070:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.154@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7484.271468] Lustre: 199070:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7484.293283] Lustre: lustre-OST0000: new disk, initializing [ 7484.297988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7484.304114] Lustre: 199094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7484.401508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7490.611225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7490.625496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7490.649894] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7490.753766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7499.773442] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7504.876854] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7505.215335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7510.443604] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7510.733947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7516.640777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7516.641277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.660385] LustreError: Skipped 1 previous similar message [ 7516.663546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.675911] Lustre: Skipped 1 previous similar message [ 7516.694162] Lustre: Skipped 1 previous similar message [ 7521.772550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.783140] Lustre: Skipped 1 previous similar message [ 7527.392147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7527.760935] Lustre: server umount lustre-OST0000 complete [ 7537.120672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.134520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.146299] Lustre: Skipped 3 previous similar messages [ 7547.366292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7547.938749] Lustre: server umount lustre-MDT0000 complete [ 7552.392757] LustreError: 197116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751343 with bad export cookie 15655038342160395161 [ 7552.408235] LustreError: 197116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7552.409144] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7553.118313] Lustre: server umount lustre-MDT0001 complete [ 7564.615808] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:49:14 (1786751354) [ 7584.233935] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7594.372548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7594.920824] LustreError: 202224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.940618] LustreError: 202224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7594.994300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7600.116132] LustreError: 202225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.796559] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7605.216080] LustreError: 202224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.205494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.339493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7610.347969] Lustre: Skipped 4 previous similar messages [ 7615.359071] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7618.161107] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7624.596764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.562567] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7640.804297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.020458] Lustre: lustre-OST0001: new disk, initializing [ 7641.024472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7641.033895] Lustre: 204531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7641.140825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7641.149722] Lustre: Skipped 2 previous similar messages [ 7647.672266] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7648.416184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7648.429381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7648.558262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7659.611816] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7667.109548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7669.739536] Lustre: DEBUG MARKER: ADD OST3 [ 7675.656331] loop: module loaded [ 7675.771409] loop0: detected capacity change from 0 to 204800000 [ 7676.011596] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7676.021046] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7676.040658] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7682.490777] loop0: detected capacity change from 0 to 204800000 [ 7683.171752] Lustre: DEBUG MARKER: STOP OST3 [ 7688.354314] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.540955] Lustre: lustre-OST0003: new disk, initializing [ 7688.545273] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7688.550725] Lustre: 206542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7690.635575] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7690.829902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7690.837808] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7690.900907] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7692.510904] Lustre: Failing over lustre-OST0003 [ 7692.603131] Lustre: server umount lustre-OST0003 complete [ 7692.706658] LustreError: 206733:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7694.739064] LustreError: 204915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.864373] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7695.871349] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7695.899376] Lustre: Skipped 2 previous similar messages [ 7696.409744] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7705.711819] Lustre: DEBUG MARKER: START OST3 again [ 7711.207309] LustreError: 203710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.241896] LustreError: 203710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7712.292242] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.531723] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7712.537122] Lustre: Skipped 1 previous similar message [ 7712.561992] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7713.904739] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7714.576970] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7714.579409] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7718.884904] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7739.521178] Lustre: Failing over lustre-OST0003 [ 7739.654203] Lustre: server umount lustre-OST0003 complete [ 7740.393208] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7740.393486] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.401778] LustreError: Skipped 1 previous similar message [ 7740.435472] Lustre: Skipped 1 previous similar message [ 7743.464435] LustreError: 203712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.484947] LustreError: 203712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7758.831721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7758.835285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.856463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7758.872543] Lustre: Skipped 3 previous similar messages [ 7765.202497] Lustre: server umount lustre-MDT0000 complete [ 7770.125066] LustreError: 202206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751561 with bad export cookie 15655038342160396043 [ 7770.131195] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.146211] LustreError: 202206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7776.718156] Lustre: server umount lustre-MDT0001 complete [ 7780.990936] Lustre: server umount lustre-OST0000 complete [ 7784.671289] Lustre: server umount lustre-OST0001 complete [ 7790.656626] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7797.655416] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7829.716831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.946818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7829.983553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7830.057544] Lustre: lustre-MDT0000: new disk, initializing [ 7830.120305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7834.045670] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7843.833103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.997928] Lustre: 211991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7844.218224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.224309] Lustre: Skipped 1 previous similar message [ 7844.252276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7848.777725] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7857.683112] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7863.657985] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7871.177177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.507698] Lustre: lustre-OST0000: new disk, initializing [ 7871.512781] Lustre: Skipped 1 previous similar message [ 7871.516689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7871.523693] Lustre: Skipped 2 previous similar messages [ 7871.530961] Lustre: 213961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7872.854149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7872.860839] Lustre: Skipped 1 previous similar message [ 7872.867793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7872.943890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7876.927626] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7885.518464] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7889.755788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7889.980606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7895.129780] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7895.347121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7898.601990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.609782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7898.617597] Lustre: Skipped 6 previous similar messages [ 7898.631938] Lustre: Skipped 8 previous similar messages [ 7911.391140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7911.596312] Lustre: server umount lustre-OST0000 complete [ 7916.000758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7921.613922] Lustre: server umount lustre-MDT0000 complete [ 7924.193293] LustreError: 213536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.206313] LustreError: 213536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 7925.620562] LustreError: 211982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751716 with bad export cookie 15655038342160398339 [ 7925.629164] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7925.637142] LustreError: 211982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7926.006801] Lustre: server umount lustre-MDT0001 complete [ 7929.730989] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:55:19 (1786751719) [ 7947.513852] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7958.005448] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7995.452599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.636416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7995.665408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7995.737513] Lustre: lustre-MDT0000: new disk, initializing [ 7995.818733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7999.631585] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8010.574381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.735892] Lustre: 218570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8010.940306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8015.341659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8025.459966] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8031.718424] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.631973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.112440] Lustre: 220537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8041.371721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8046.991186] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8056.386905] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.948891] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8061.196100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8066.223758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8066.425689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8072.160800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8072.166383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.174797] LustreError: Skipped 1 previous similar message [ 8072.184774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.197024] Lustre: Skipped 3 previous similar messages [ 8072.209137] Lustre: Skipped 9 previous similar messages [ 8074.977993] Lustre: server umount lustre-OST0000 complete [ 8084.976656] Lustre: server umount lustre-MDT0000 complete [ 8087.521703] LustreError: 218582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.542621] LustreError: 218582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8089.482864] LustreError: 218562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751880 with bad export cookie 15655038342160399221 [ 8089.495742] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8089.918889] Lustre: server umount lustre-MDT0001 complete [ 8098.509231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.298183] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8113.908823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8114.470755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8114.478325] Lustre: Skipped 5 previous similar messages [ 8119.980386] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8128.472470] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.368844] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.938732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.045282] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8158.664095] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8169.450602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.840488] Lustre: server umount lustre-OST0000 complete [ 8185.074899] Lustre: server umount lustre-MDT0000 complete [ 8188.710473] LustreError: 222733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751979 with bad export cookie 15655038342160400103 [ 8188.734035] LustreError: 222733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8188.735100] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.128777] Lustre: server umount lustre-MDT0001 complete [ 8299.499130] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8301.812524] Key type lgssc unregistered [ 8302.070661] LNet: 226434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8302.081381] LNetError: 226434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8303.141882] LNet: Removed LNI 192.168.202.154@tcp [ 8304.217170] Key type .llcrypt unregistered [ 8304.224254] Key type ._llcrypt unregistered [ 8331.384788] Key type ._llcrypt registered [ 8331.387507] Key type .llcrypt registered [ 8331.568363] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 8348.337258] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8350.157808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8350.313164] alg: No test for adler32 (adler32-zlib) [ 8351.687679] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 8352.081266] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8353.879663] Key type lgssc registered [ 8355.528213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8393.502584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8393.526668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8394.738412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8394.782344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8394.858340] Lustre: lustre-MDT0000: new disk, initializing [ 8394.983366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.019218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8398.995182] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8411.385886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8411.559957] Lustre: 229786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8411.646866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8411.653689] Lustre: Skipped 1 previous similar message [ 8411.838879] Lustre: lustre-MDT0001: new disk, initializing [ 8411.997532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8412.072671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8412.092068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8415.971448] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8426.486770] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8433.416937] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8441.818021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8442.116512] Lustre: lustre-OST0000: new disk, initializing [ 8442.120886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8442.125702] Lustre: 231758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8442.205827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.074594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8448.084864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8448.261086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8448.910374] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8459.587641] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8464.055443] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8464.317465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8468.760673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8468.935941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8474.080572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8474.087069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.097628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8474.105327] Lustre: Skipped 1 previous similar message [ 8474.119617] Lustre: Skipped 1 previous similar message [ 8476.722714] Lustre: server umount lustre-OST0000 complete [ 8483.808937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8483.825787] LustreError: Skipped 1 previous similar message [ 8483.831484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.848667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.443896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.446785] Lustre: Skipped 3 previous similar messages [ 8494.561588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8494.573038] Lustre: Skipped 1 previous similar message [ 8496.095588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8496.467370] Lustre: server umount lustre-MDT0000 complete [ 8500.726679] LustreError: 229781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752291 with bad export cookie 2848202827797387533 [ 8500.735108] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8500.737140] LustreError: 229781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8500.994683] Lustre: server umount lustre-MDT0001 complete [ 8522.958965] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8533.714343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.016296] LustreError: 234715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.032469] LustreError: 234715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8534.069317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8538.224717] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8539.105784] LustreError: 234716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.229082] LustreError: 234715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.580420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.924723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8554.127448] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8557.929296] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8564.143741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8564.366935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.876443] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8581.235866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.425422] Lustre: lustre-OST0001: new disk, initializing [ 8581.427714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8581.431890] Lustre: 237025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.537459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8587.992855] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8589.948625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8589.954791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8590.050805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8600.032950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8607.028151] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8620.472708] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:06:50 (1786752410) [ 8624.618782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8624.627958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.644625] Lustre: Skipped 1 previous similar message [ 8624.650236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.897958] Lustre: server umount lustre-MDT0000 complete [ 8631.291460] LustreError: 234715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8631.317560] LustreError: 234715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8633.983438] LustreError: 236221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752425 with bad export cookie 2848202827797388415 [ 8633.990444] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.995508] LustreError: 236221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8634.363920] Lustre: server umount lustre-MDT0001 complete [ 8650.217689] Lustre: server umount lustre-OST0000 complete [ 8652.133117] Lustre: 228012:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752427/real 1786752427] req@ffff9364c2701180 x1873545415211520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786752443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8652.151941] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.159125] Lustre: Skipped 3 previous similar messages [ 8653.791174] Lustre: 228010:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752429/real 1786752429] req@ffff9364c75a1180 x1873545415211776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786752445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8655.020344] Lustre: server umount lustre-OST0001 complete [ 8661.784372] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 8672.710449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8710.672128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.898658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8710.925662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8710.992294] Lustre: lustre-MDT0000: new disk, initializing [ 8711.077824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8711.102954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8716.587543] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8726.829597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8726.938574] Lustre: 240971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8726.985454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8726.988442] Lustre: Skipped 1 previous similar message [ 8727.040885] Lustre: lustre-MDT0001: new disk, initializing [ 8727.136575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8727.148199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8730.872736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8740.926959] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.724456] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.755591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8755.044053] Lustre: lustre-OST0000: new disk, initializing [ 8755.046976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8755.052383] Lustre: 242939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8756.576699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8756.586900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.758731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8760.633333] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8770.091923] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8775.336750] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8775.670029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8781.386643] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8781.679724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8787.437138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8787.438513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.461867] LustreError: Skipped 1 previous similar message [ 8787.471420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8787.487130] Lustre: Skipped 1 previous similar message [ 8787.492753] Lustre: Skipped 4 previous similar messages [ 8798.175475] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8798.361955] Lustre: server umount lustre-OST0000 complete [ 8802.788827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.814751] Lustre: Skipped 1 previous similar message [ 8807.907530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.918100] Lustre: Skipped 8 previous similar messages [ 8808.701336] Lustre: server umount lustre-MDT0000 complete [ 8812.549090] LustreError: 242950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752603 with bad export cookie 2848202827797390550 [ 8812.553908] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.560621] LustreError: 242950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8813.087465] Lustre: server umount lustre-MDT0001 complete [ 8821.507562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.874147] LustreError: 245157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.997242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8822.007794] Lustre: Skipped 2 previous similar messages [ 8825.634642] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8832.485882] LustreError: 245158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.508937] LustreError: 245158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8834.020158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.820772] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8845.901698] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8851.053316] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8858.186164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8864.886137] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8872.026398] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9019.371415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.395819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.421828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9025.481943] Lustre: server umount lustre-OST0000 complete [ 9035.939231] Lustre: server umount lustre-MDT0000 complete [ 9036.769169] LustreError: 245152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.806679] LustreError: 245152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9039.861189] LustreError: 247002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752831 with bad export cookie 2848202827797391432 [ 9039.875332] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9039.879483] LustreError: 247002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9040.305599] Lustre: server umount lustre-MDT0001 complete [ 9139.634257] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9143.359752] Key type lgssc unregistered [ 9143.821601] LNet: 248768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9143.834742] LNetError: 248768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.870727] LNet: Removed LNI 192.168.202.154@tcp [ 9145.955143] Key type .llcrypt unregistered [ 9145.959360] Key type ._llcrypt unregistered [ 9176.348673] Key type ._llcrypt registered [ 9176.354838] Key type .llcrypt registered [ 9176.563904] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9195.248758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9196.515982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9196.559047] alg: No test for adler32 (adler32-zlib) [ 9198.101967] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 9198.530097] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9200.343176] Key type lgssc registered [ 9202.016952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9254.368916] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9264.160743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9264.206959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.566592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9265.626282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9265.756045] Lustre: lustre-MDT0000: new disk, initializing [ 9265.927394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.956457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9270.327861] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9281.142180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.369520] Lustre: 252841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9281.507881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9281.523986] Lustre: Skipped 1 previous similar message [ 9281.620482] Lustre: lustre-MDT0001: new disk, initializing [ 9281.726154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9281.782100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9281.803171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9286.565095] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9292.649512] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9301.385268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9302.076064] Lustre: lustre-OST0000: new disk, initializing [ 9302.089025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9302.100385] Lustre: 254470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9302.304696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9308.868997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9308.881658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9309.025341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9309.908459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9319.934761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.207640] Lustre: lustre-OST0001: new disk, initializing [ 9320.211261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9320.218182] Lustre: 255344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9320.313897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9326.471955] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9326.729274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9326.741429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9326.928278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9337.670692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9346.823865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9358.817296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9358.822261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.957552] Lustre: server umount lustre-MDT0000 complete [ 9362.861066] LustreError: 252834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753154 with bad export cookie 272906737274801885 [ 9362.871796] LustreError: 252834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9362.876321] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9362.923065] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.926598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9362.955249] Lustre: Skipped 2 previous similar messages [ 9362.989288] Lustre: Skipped 1 previous similar message [ 9363.793744] Lustre: server umount lustre-MDT0001 complete [ 9370.350257] Lustre: server umount lustre-OST0000 complete [ 9374.724507] Lustre: server umount lustre-OST0001 complete [ 9388.316570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9396.263391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9403.173714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9411.869944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9433.012985] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9445.409328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.425925] Lustre: lustre-MDT0000: reset Object Index mappings [ 9445.801416] LustreError: 260354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9445.827804] LustreError: 260354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9445.874897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.570468] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9450.979596] LustreError: 260355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.106363] LustreError: 260354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.877049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.892977] Lustre: lustre-MDT0001: reset Object Index mappings [ 9460.215865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9467.579911] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9471.797930] Lustre: 261496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9479.643812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9479.936414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9485.035970] LustreError: 261852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.988850] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9490.405341] LustreError: 261852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.463224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9494.398766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9500.165802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9502.328453] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9510.225794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9514.162058] Lustre: 263368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9521.646713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9521.670789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.694295] Lustre: Skipped 1 previous similar message [ 9521.701113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9525.740660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9525.747366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9525.761033] Lustre: Skipped 1 previous similar message [ 9525.782176] Lustre: Skipped 3 previous similar messages [ 9527.229114] Lustre: server umount lustre-MDT0000 complete [ 9530.852726] LustreError: 261277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9530.875190] LustreError: 261277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9531.197610] LustreError: 260336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753322 with bad export cookie 272906737274808668 [ 9531.205420] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9531.216169] LustreError: 260336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9531.546858] Lustre: server umount lustre-MDT0001 complete [ 9545.289862] Lustre: server umount lustre-OST0000 complete [ 9559.238525] Lustre: server umount lustre-OST0001 complete [ 9579.523319] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:22:49 (1786753369) [ 9595.705320] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9605.706787] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9639.400669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.664823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9639.742099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9639.866225] Lustre: lustre-MDT0000: new disk, initializing [ 9639.974235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.978406] Lustre: Skipped 1 previous similar message [ 9640.006560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9643.545924] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9654.047410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.180364] Lustre: 267622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9654.254148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9654.266633] Lustre: Skipped 1 previous similar message [ 9654.404965] Lustre: lustre-MDT0001: new disk, initializing [ 9654.566027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9654.586640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9658.179212] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9669.936192] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9675.361577] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9683.714179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9684.158648] Lustre: lustre-OST0000: new disk, initializing [ 9684.170737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9684.189743] Lustre: 269590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9686.013961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9686.027388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9686.233974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9693.389843] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9705.877628] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9711.884316] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9712.037231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9717.278407] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9717.495433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9721.828225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9721.839060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.855473] Lustre: Skipped 1 previous similar message [ 9721.860710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9725.327191] Lustre: server umount lustre-OST0000 complete [ 9731.554515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9731.563720] LustreError: Skipped 1 previous similar message [ 9731.577359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.596164] Lustre: Skipped 1 previous similar message [ 9731.600723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.613931] Lustre: Skipped 2 previous similar messages [ 9735.711275] Lustre: server umount lustre-MDT0000 complete [ 9737.699200] LustreError: 267630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.714775] LustreError: 267630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9738.993301] LustreError: 267615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753530 with bad export cookie 272906737274810768 [ 9739.004683] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9739.009506] LustreError: 267615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9739.024383] LustreError: 250348:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 9739.351969] Lustre: server umount lustre-MDT0001 complete [ 9756.828940] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9768.395476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.970024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.978837] Lustre: Skipped 2 previous similar messages [ 9774.066900] LustreError: 272528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.105292] LustreError: 272528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9775.002663] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9786.094684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.427113] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9795.162059] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9802.497721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9807.873234] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9816.356935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9816.570193] Lustre: lustre-OST0001: new disk, initializing [ 9816.573809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9816.580267] Lustre: 274834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9817.850984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9817.865158] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9817.976345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9823.596967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9834.301530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9838.405784] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9843.894111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9844.082715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9852.906699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.921313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.943193] Lustre: Skipped 1 previous similar message [ 9852.960758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.972048] Lustre: Skipped 2 previous similar messages [ 9856.593955] Lustre: server umount lustre-MDT0000 complete [ 9859.046706] LustreError: 272524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.064051] LustreError: 272524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9860.179427] LustreError: 272507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753651 with bad export cookie 272906737274811650 [ 9860.210103] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9860.214513] LustreError: 272507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9860.500218] Lustre: server umount lustre-MDT0001 complete [ 9874.675372] Lustre: server umount lustre-OST0000 complete [ 9888.040191] Lustre: server umount lustre-OST0001 complete [ 9909.075811] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9923.165786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9923.777872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9923.782921] Lustre: Skipped 3 previous similar messages [ 9929.297439] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9937.628878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9941.520667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9944.389966] Lustre: 279034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9951.341724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9958.062688] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9966.138795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9968.385434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9968.402469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9972.178600] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9978.942551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9982.574686] Lustre: 280909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9988.033556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9988.145378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9996.251995] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:29:46 (1786753786) [10014.715050] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10014.720439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10014.727891] Lustre: Skipped 6 previous similar messages [10014.732042] Lustre: Skipped 6 previous similar messages [10017.272867] Lustre: server umount lustre-MDT0000 complete [10019.820230] LustreError: 277889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.849687] LustreError: 277889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10021.665585] LustreError: 280911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753812 with bad export cookie 272906737274818202 [10021.675114] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10021.677155] LustreError: 280911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10028.371509] Lustre: server umount lustre-MDT0001 complete [10039.182322] Lustre: server umount lustre-OST0000 complete [10048.993683] Lustre: server umount lustre-OST0001 complete [10055.313697] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [10066.013617] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10113.789724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10124.735806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.061809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10125.100770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10125.206634] Lustre: lustre-MDT0000: new disk, initializing [10125.327238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10128.913596] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10139.923814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.028221] Lustre: 285152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10140.229224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10144.978745] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10158.409368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.899341] Lustre: 286786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10160.804362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10165.906802] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10176.973330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10177.178846] Lustre: 287655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10178.772339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10178.786183] Lustre: Skipped 1 previous similar message [10178.856142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10184.289735] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10194.646890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10198.926881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10198.933664] Lustre: Skipped 1 previous similar message [10214.252343] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 20:33:24 (1786754004) [10216.929781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10216.947227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.969707] Lustre: Skipped 2 previous similar messages [10216.975905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10216.979801] Lustre: Skipped 5 previous similar messages [10219.226451] Lustre: server umount lustre-MDT0000 complete [10223.347562] LustreError: 286785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754014 with bad export cookie 272906737274825846 [10223.351973] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10223.367608] LustreError: 286785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10223.744445] Lustre: server umount lustre-MDT0001 complete [10238.647148] Lustre: server umount lustre-OST0000 complete [10253.422516] Lustre: server umount lustre-OST0001 complete [10263.598354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.460664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.479462] Lustre: Skipped 7 previous similar messages [10269.461533] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10278.900122] LustreError: 289903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.938222] LustreError: 289903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10279.392040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.115189] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10291.362431] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10296.803555] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10303.384408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.938120] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10318.279308] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10334.442167] loop1: detected capacity change from 0 to 204800000 [10334.714360] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10334.728326] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10334.746948] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10339.163371] loop1: detected capacity change from 0 to 204800000 [10344.679234] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.783040] LustreError: 289892:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6389: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10344.794571] LustreError: 289892:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6389 [10344.801369] LustreError: 289892:0:(mgs_handler.c:517:mgs_target_reg()) fs6389-MDT0001: Error checking target index: rc = -13 [10344.815608] LustreError: fs6389-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10344.825526] LustreError: 293621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10344.832519] LustreError: 293621:0:(tgt_mount.c:2129:server_put_super()) no obd fs6389-MDT0001 [10344.840964] LustreError: 293621:0:(tgt_mount.c:116:server_deregister_mount()) fs6389-MDT0001 not registered [10344.886528] Lustre: server umount fs6389-MDT0001 complete [10344.889412] LustreError: 293621:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10353.591947] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.647206] Lustre: Setting parameter fs6389-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6389-MDT0001 [10353.721264] Lustre: srv-fs6389-MDT0001: No data found on store. Initialize space. [10353.724723] Lustre: Skipped 4 previous similar messages [10353.802344] Lustre: fs6389-MDT0001: new disk, initializing [10353.805788] Lustre: Skipped 3 previous similar messages [10353.980589] Lustre: srv-fs6389-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10353.987728] Lustre: 294089:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6389-MDT0001-mdtlov: can't create new llog: rc = -115 [10353.997713] LustreError: 294089:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6389-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10357.663095] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10365.187793] Lustre: server umount fs6389-MDT0001 complete [10379.120211] loop1: detected capacity change from 0 to 204800000 [10385.781988] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10386.726612] Lustre: 295657:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6389-MDT0001-mdtlov: can't create new llog: rc = -115 [10386.735582] LustreError: 295657:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6389-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10391.547803] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10393.125224] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10398.470371] Lustre: server umount fs6389-MDT0001 complete [10407.906546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.920931] Lustre: Skipped 3 previous similar messages [10407.925698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10413.596449] Lustre: server umount lustre-MDT0000 complete [10421.455382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.533333] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.065787] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10426.888552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10433.939556] loop1: detected capacity change from 0 to 204800000 [10439.705302] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.212116] Lustre: 298102:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6389-MDT0001-mdtlov: can't create new llog: rc = -115 [10440.233274] LustreError: 298102:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6389-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10445.271780] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10449.303733] Lustre: server umount fs6389-MDT0001 complete [10469.346322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10469.359034] LustreError: Skipped 2 previous similar messages [10482.146132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10482.374425] Lustre: server umount lustre-OST0000 complete [10492.246366] Lustre: server umount lustre-MDT0000 complete [10495.707452] LustreError: 289884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754286 with bad export cookie 272906737274828527 [10495.709418] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.716742] LustreError: 289884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10496.074614] Lustre: server umount lustre-MDT0001 complete [10507.288908] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10510.756050] Key type lgssc unregistered [10511.189381] LNet: 300450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10511.218628] LNetError: 300450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10511.254251] LNet: Removed LNI 192.168.202.154@tcp [10512.775503] Key type .llcrypt unregistered [10512.777590] Key type ._llcrypt unregistered [10533.680989] Key type ._llcrypt registered [10533.683376] Key type .llcrypt registered [10533.818745] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10534.776522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10534.888197] alg: No test for adler32 (adler32-zlib) [10536.270202] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [10536.490891] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10538.239265] Key type lgssc registered [10539.336835] Lustre: Echo OBD driver; http://www.lustre.org/ [10546.869980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10546.905579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.673677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.659197] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10557.926606] LustreError: 301720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.957971] LustreError: 301720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10561.106081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.395799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10566.016328] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10574.833711] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10581.484849] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10589.646123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10590.230181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10593.340710] LustreError: 303550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10596.717522] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10598.389520] LustreError: 303740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10603.498926] LustreError: 303551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.741890] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10609.800397] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.827430] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10620.275539] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.297161] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10625.338957] loop1: detected capacity change from 0 to 204800000 [10625.553428] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10625.570118] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10625.583780] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10630.953165] loop1: detected capacity change from 0 to 204800000 [10636.844743] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10636.950488] Lustre: fs32274-OST0000: new disk, initializing [10636.953504] Lustre: srv-fs32274-OST0000: No data found on store. Initialize space. [10636.958109] Lustre: 305454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10637.018913] Lustre: fs32274-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10637.286174] LustreError: 303551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.311383] LustreError: 303551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10644.913134] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10653.994438] Lustre: server umount fs32274-OST0000 complete [10666.957591] loop1: detected capacity change from 0 to 204800000 [10671.470593] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.480720] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10672.366922] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10672.540737] Lustre: fs32274-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10673.124060] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10679.601241] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10688.513629] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10693.367907] loop2: detected capacity change from 0 to 204800000 [10693.574196] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.645077] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.672408] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10700.775464] loop2: detected capacity change from 0 to 204800000 [10706.516792] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10706.610737] LustreError: fs32274-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10706.623533] LustreError: fs32274-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10706.638739] LustreError: 308497:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10706.652657] LustreError: 308497:0:(tgt_mount.c:2129:server_put_super()) no obd fs32274-OST0001 [10706.663373] LustreError: 308497:0:(tgt_mount.c:116:server_deregister_mount()) fs32274-OST0001 not registered [10706.709944] Lustre: server umount fs32274-OST0001 complete [10706.713667] LustreError: 308497:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10716.154242] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.534730] Lustre: fs32274-OST0001: new disk, initializing [10716.544832] Lustre: srv-fs32274-OST0001: No data found on store. Initialize space. [10716.554163] Lustre: 309048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10716.672742] Lustre: fs32274-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10722.412594] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10730.295714] Lustre: server umount fs32274-OST0000 complete [10738.041080] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.052952] LustreError: 303552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10739.254606] Lustre: server umount fs32274-OST0001 complete [10752.271499] loop1: detected capacity change from 0 to 204800000 [10757.425028] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10757.605874] Lustre: fs32274-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.116422] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10772.398921] loop2: detected capacity change from 0 to 204800000 [10777.527228] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10784.621505] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10796.717829] Lustre: server umount fs32274-OST0000 complete [10809.444337] Lustre: server umount fs32274-OST0001 complete [10826.722640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.732439] Lustre: Skipped 1 previous similar message [10826.736626] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10826.738365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10826.754379] Lustre: Skipped 1 previous similar message [10830.787883] Lustre: server umount lustre-OST0000 complete [10836.992410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10837.017166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10840.829583] Lustre: server umount lustre-MDT0000 complete [10845.517316] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754636 with bad export cookie 9682659088075391911 [10845.526930] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.544511] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10846.147173] Lustre: server umount lustre-MDT0001 complete [10859.516658] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10862.240916] Key type lgssc unregistered [10862.489124] LNet: 314974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10862.493536] LNetError: 314974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10862.505612] LNet: Removed LNI 192.168.202.154@tcp [10863.349323] Key type .llcrypt unregistered [10863.353508] Key type ._llcrypt unregistered [10897.768839] Key type ._llcrypt registered [10897.771389] Key type .llcrypt registered [10897.880937] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [10912.470464] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10913.797596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10913.855570] alg: No test for adler32 (adler32-zlib) [10915.165436] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [10915.406027] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10917.063158] Key type lgssc registered [10918.518702] Lustre: Echo OBD driver; http://www.lustre.org/ [10953.055400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10953.078093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.396805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10954.425821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10954.495869] Lustre: lustre-MDT0000: new disk, initializing [10954.559810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10954.575605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10959.085613] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10969.517333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.710627] Lustre: 318834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10969.761454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10969.764849] Lustre: Skipped 1 previous similar message [10969.857542] Lustre: lustre-MDT0001: new disk, initializing [10970.079187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10970.125711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10970.140576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10974.807730] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10984.799135] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.946633] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10996.418126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10996.816572] Lustre: lustre-OST0000: new disk, initializing [10996.828607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10996.837499] Lustre: 320804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10996.943419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11002.977142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11003.000623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11003.203903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11003.708182] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11013.186315] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11015.459904] Lustre: Mounted lustre-client - version 2.17.57_1_g59aa375 [11016.453799] Lustre: Unmounted lustre-client [11025.900324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.920161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11025.925698] Lustre: Skipped 1 previous similar message [11031.012464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11031.040413] Lustre: Skipped 1 previous similar message [11031.343343] Lustre: server umount lustre-OST0000 complete [11036.131242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.134216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11036.156157] Lustre: Skipped 2 previous similar messages [11036.173222] Lustre: Skipped 1 previous similar message [11041.249509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11041.253557] Lustre: Skipped 1 previous similar message [11041.635857] Lustre: server umount lustre-MDT0000 complete [11045.366962] LustreError: 318827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754836 with bad export cookie 13370588375531754174 [11045.389514] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.687237] Lustre: server umount lustre-MDT0001 complete [11054.362978] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11057.090658] Key type lgssc unregistered [11071.647881] LNet: 322966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11071.657773] LNetError: 322966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11146.406295] LNet: Removed LNI 192.168.202.154@tcp [11147.167807] Key type .llcrypt unregistered [11147.180300] Key type ._llcrypt unregistered [11167.090140] Key type ._llcrypt registered [11167.095790] Key type .llcrypt registered [11167.235705] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11182.150253] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11183.661371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11183.668635] alg: No test for adler32 (adler32-zlib) [11184.794641] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11185.084402] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11186.759885] Key type lgssc registered [11187.942898] Lustre: Echo OBD driver; http://www.lustre.org/ [11215.433480] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:50:05 (1786755005) [11221.639750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11221.673028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.886423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11222.913787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11222.979322] Lustre: lustre-MDT0000: new disk, initializing [11223.049120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11223.066820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11228.844672] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11241.851702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.037614] Lustre: 326195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11242.123184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11242.137374] Lustre: Skipped 1 previous similar message [11242.260106] Lustre: lustre-MDT0001: new disk, initializing [11242.445801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11242.486672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11242.504333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11246.605494] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11255.310722] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11261.999024] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11268.676063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11268.925423] Lustre: lustre-OST0000: new disk, initializing [11268.928533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11268.934852] Lustre: 328170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11269.028374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11274.689434] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11275.941367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11275.959372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11276.120326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11283.784737] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.865198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11301.867183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11301.883277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11301.891629] Lustre: Skipped 1 previous similar message [11303.161413] Lustre: server umount lustre-OST0000 complete [11306.979222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.987604] Lustre: Skipped 1 previous similar message [11306.993369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11306.999027] Lustre: Skipped 1 previous similar message [11312.100825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11312.118236] Lustre: Skipped 1 previous similar message [11317.218670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.226984] Lustre: Skipped 1 previous similar message [11321.311159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11321.465523] Lustre: server umount lustre-MDT0000 complete [11325.614538] LustreError: 326187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755116 with bad export cookie 11730446524818013111 [11325.624051] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11325.627737] LustreError: 326187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11325.994433] Lustre: server umount lustre-MDT0001 complete [11339.925684] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11342.701612] Key type lgssc unregistered [11342.921297] LNet: 330081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11342.931630] LNetError: 330081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11342.955173] LNet: Removed LNI 192.168.202.154@tcp [11344.124407] Key type .llcrypt unregistered [11344.131896] Key type ._llcrypt unregistered [11366.774097] Key type ._llcrypt registered [11366.780884] Key type .llcrypt registered [11366.883943] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11367.972196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11368.325678] alg: No test for adler32 (adler32-zlib) [11369.656317] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11369.930890] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11371.591214] Key type lgssc registered [11372.824367] Lustre: Echo OBD driver; http://www.lustre.org/ [11379.868389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11379.887562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11381.394624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11385.404196] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11390.961199] LustreError: 331356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.984804] LustreError: 331356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11395.548515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.979462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11400.324469] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11407.789695] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11413.842799] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11420.467188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11420.809912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11427.223034] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11435.251975] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11447.411170] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11449.230462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11457.009646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11457.014735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11457.019735] Lustre: Skipped 2 previous similar messages [11457.027617] Lustre: Skipped 2 previous similar messages [11458.471578] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [11462.124301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11467.235682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11467.246416] Lustre: Skipped 3 previous similar messages [11468.255145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11468.653960] LustreError: 331352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.902962] Lustre: server umount lustre-MDT0000 complete [11472.360087] LustreError: 331356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.396176] LustreError: 331356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11473.662824] LustreError: 331338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755264 with bad export cookie 1694279354259064804 [11473.670311] LustreError: 331338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11473.674302] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11473.783147] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [11477.472692] LustreError: 331352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11477.477791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.505121] LustreError: 331352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11480.355534] Lustre: server umount lustre-MDT0001 complete [11485.533538] Lustre: server umount lustre-OST0000 complete [11494.518714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.285904] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11502.440450] Lustre: 335537:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.154@tcp [11504.978654] Lustre: server umount MGS complete [11513.541490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.982228] LustreError: 336108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.012094] LustreError: 336108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11514.116095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.519498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11524.592046] LustreError: 336109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11524.637487] LustreError: 336109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11526.845560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11527.217919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11531.247084] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11540.333162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11540.883312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11546.533204] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11554.847953] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11556.387468] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11562.286429] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11566.179815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11572.708587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11572.714576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.734832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11572.739819] Lustre: Skipped 3 previous similar messages [11573.903631] Lustre: server umount lustre-OST0000 complete [11579.875470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.886734] Lustre: Skipped 1 previous similar message [11588.172747] Lustre: server umount lustre-MDT0000 complete [11589.096941] LustreError: 336105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.107138] LustreError: 336105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11592.366437] LustreError: 336090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755383 with bad export cookie 1694279354259065644 [11592.374802] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.375875] LustreError: 336090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11592.685987] Lustre: server umount lustre-MDT0001 complete [11607.881652] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11611.615601] Key type lgssc unregistered [11611.919618] LNet: 339547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11611.928689] LNetError: 339547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11611.957791] LNet: Removed LNI 192.168.202.154@tcp [11612.755478] Key type .llcrypt unregistered [11612.757641] Key type ._llcrypt unregistered [11630.324939] Key type ._llcrypt registered [11630.328256] Key type .llcrypt registered [11630.447790] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11645.474430] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11646.926165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11646.988670] alg: No test for adler32 (adler32-zlib) [11648.027300] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [11648.231383] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11649.881932] Key type lgssc registered [11650.878336] Lustre: Echo OBD driver; http://www.lustre.org/ [11683.827840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11683.891960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.104808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11685.147365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11685.207560] Lustre: lustre-MDT0000: new disk, initializing [11685.274483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11685.291710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11689.219370] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11699.379751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.484446] Lustre: 342707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11699.539482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11699.543104] Lustre: Skipped 1 previous similar message [11699.620588] Lustre: lustre-MDT0001: new disk, initializing [11699.720452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11699.764418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11699.778427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11703.929215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11711.489191] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11716.958401] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11724.564126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11724.875740] Lustre: lustre-OST0000: new disk, initializing [11724.885507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11724.899178] Lustre: 344679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11725.054609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11731.477854] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11731.586467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11731.606026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11731.742665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11742.717831] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11748.331276] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11748.640368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11753.002646] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11753.190856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11757.536537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11757.551196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.553373] LustreError: Skipped 1 previous similar message [11757.558926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11757.588876] Lustre: Skipped 1 previous similar message [11761.568254] Lustre: server umount lustre-OST0000 complete [11766.240442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11766.249471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11766.290061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11766.292790] Lustre: Skipped 1 previous similar message [11767.778183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11767.778979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.548158] Lustre: server umount lustre-MDT0000 complete [11776.947275] LustreError: 342702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755568 with bad export cookie 18297650483406454392 [11776.954414] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11776.960413] LustreError: 342702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11777.242948] Lustre: server umount lustre-MDT0001 complete [11788.259625] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:59:38 (1786755578) [11801.869912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11811.990888] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11842.559318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.831308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11842.868781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11842.923878] Lustre: lustre-MDT0000: new disk, initializing [11842.976408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.988302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11846.976674] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11856.168978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.286650] Lustre: 349436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11856.366643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11856.376936] Lustre: Skipped 1 previous similar message [11856.499457] Lustre: lustre-MDT0001: new disk, initializing [11856.635650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11856.665544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11856.680227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11859.980504] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11868.585285] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11874.503956] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11881.911443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11882.113579] Lustre: lustre-OST0000: new disk, initializing [11882.117396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11882.123407] Lustre: 351415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11882.235903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11889.230877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11889.246256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11889.398413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11889.959511] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11899.378238] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11930.603252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.610672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11930.630711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11930.630720] Lustre: Skipped 1 previous similar message [11934.799641] Lustre: server umount lustre-OST0000 complete [11938.784826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11938.787169] LustreError: Skipped 1 previous similar message [11938.789704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.796183] Lustre: Skipped 1 previous similar message [11938.799476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.802508] Lustre: Skipped 1 previous similar message [11944.588184] Lustre: server umount lustre-MDT0000 complete [11945.952876] LustreError: 349449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.977405] LustreError: 349449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11948.164337] LustreError: 349428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755739 with bad export cookie 18297650483406455274 [11948.172949] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11948.432237] Lustre: server umount lustre-MDT0001 complete [11957.488292] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11960.020375] Key type lgssc unregistered [11960.280462] LNet: 353569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11960.284276] LNetError: 353569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11960.294838] LNet: Removed LNI 192.168.202.154@tcp [11960.957774] Key type .llcrypt unregistered [11960.959912] Key type ._llcrypt unregistered [12043.645790] Key type ._llcrypt registered [12043.647540] Key type .llcrypt registered [12043.737844] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12044.085570] Key type .llcrypt unregistered [12044.091151] Key type ._llcrypt unregistered [12107.271267] Key type ._llcrypt registered [12107.277457] Key type .llcrypt registered [12107.373835] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12107.638431] Key type .llcrypt unregistered [12107.640868] Key type ._llcrypt unregistered [12133.154936] Key type ._llcrypt registered [12133.163184] Key type .llcrypt registered [12133.271809] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12133.517704] Key type .llcrypt unregistered [12133.526898] Key type ._llcrypt unregistered [12164.949498] Key type ._llcrypt registered [12164.951427] Key type .llcrypt registered [12165.037076] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12165.260365] Key type .llcrypt unregistered [12165.262354] Key type ._llcrypt unregistered [12236.748446] Key type ._llcrypt registered [12236.750288] Key type .llcrypt registered [12236.861923] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12237.139444] Key type .llcrypt unregistered [12237.141501] Key type ._llcrypt unregistered [12263.130470] Key type ._llcrypt registered [12263.132231] Key type .llcrypt registered [12263.238449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12263.480284] Key type .llcrypt unregistered [12263.481981] Key type ._llcrypt unregistered [12290.525448] Key type ._llcrypt registered [12290.528775] Key type .llcrypt registered [12290.644482] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12290.969361] Key type .llcrypt unregistered [12290.975775] Key type ._llcrypt unregistered [12324.421078] Key type ._llcrypt registered [12324.423304] Key type .llcrypt registered [12324.505734] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12324.804596] Key type .llcrypt unregistered [12324.809719] Key type ._llcrypt unregistered [12353.019034] Key type ._llcrypt registered [12353.021303] Key type .llcrypt registered [12353.164152] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12353.581562] Key type .llcrypt unregistered [12353.584278] Key type ._llcrypt unregistered [12379.869087] Key type ._llcrypt registered [12379.875970] Key type .llcrypt registered [12379.972749] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12380.231161] Key type .llcrypt unregistered [12380.236214] Key type ._llcrypt unregistered [12441.233523] Key type ._llcrypt registered [12441.235212] Key type .llcrypt registered [12441.320827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12441.581417] Key type .llcrypt unregistered [12441.583940] Key type ._llcrypt unregistered [12478.351372] Key type ._llcrypt registered [12478.356060] Key type .llcrypt registered [12478.509917] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12478.937447] Key type .llcrypt unregistered [12478.940420] Key type ._llcrypt unregistered [12513.263130] Key type ._llcrypt registered [12513.264953] Key type .llcrypt registered [12513.366091] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12513.615618] Key type .llcrypt unregistered [12513.621073] Key type ._llcrypt unregistered [12582.352754] Key type ._llcrypt registered [12582.362102] Key type .llcrypt registered [12582.731979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12598.651526] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12599.919529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12600.069950] alg: No test for adler32 (adler32-zlib) [12601.165319] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [12601.487408] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12603.191196] Key type lgssc registered [12604.311332] Lustre: Echo OBD driver; http://www.lustre.org/ [12638.134912] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12638.150667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.563877] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12639.605455] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12639.689530] Lustre: tst250fs-MDT0000: new disk, initializing [12639.796442] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12639.827177] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12644.508790] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12656.971928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.158287] Lustre: 373807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12657.236637] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12657.242387] Lustre: Skipped 1 previous similar message [12657.350373] Lustre: tst250fs-MDT0001: new disk, initializing [12657.500069] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12657.546651] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12657.567273] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12664.376167] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12670.871573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12678.814841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12679.187431] Lustre: tst250fs-OST0000: new disk, initializing [12679.190911] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12679.195537] Lustre: 375441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12679.278895] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12680.441386] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12680.461393] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12680.668350] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12686.249306] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12698.048698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12698.282813] Lustre: tst250fs-OST0001: new disk, initializing [12698.286287] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12698.295990] Lustre: 376311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12698.409822] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12705.718322] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12707.415653] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12707.426377] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12707.550863] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12711.096956] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12713.225573] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12713.235973] Lustre: Skipped 1 previous similar message [12738.532887] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.541421] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12738.554771] Lustre: Skipped 3 previous similar messages [12738.564449] Lustre: Skipped 3 previous similar messages [12740.844939] Lustre: server umount tst250fs-MDT0000 complete [12744.375136] LustreError: 373798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756535 with bad export cookie 12548097904909573871 [12744.378640] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12744.395748] LustreError: 373798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12744.743272] Lustre: server umount tst250fs-MDT0001 complete [12758.174944] Lustre: server umount tst250fs-OST0000 complete [12771.142888] Lustre: server umount tst250fs-OST0001 complete [12789.883755] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12799.140103] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12831.134993] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 21:17:01 (1786756621) [12832.574481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12850.910361] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12861.534938] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12896.652636] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12574 sec ========= 21:18:06 (1786756686) [12899.031497] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:18:08 (1786756688) === [12902.352645] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:18:12 (1786756692) === [12940.706817] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12943.391454] Key type lgssc unregistered [12943.674488] LNet: 385459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12943.686457] LNetError: 385459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12943.704821] LNet: Removed LNI 192.168.202.154@tcp [12944.643501] Key type .llcrypt unregistered [12944.646500] Key type ._llcrypt unregistered