[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 551309178 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002387] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009142] LSM: Security Framework initializing [ 0.010054] Yama: becoming mindful. [ 0.012021] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.022124] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026805] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] *** VALIDATE tmpfs *** [ 0.030506] *** VALIDATE proc *** [ 0.032234] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035115] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042277] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.045171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046750] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054251] rcu: Hierarchical SRCU implementation. [ 0.056343] smp: Bringing up secondary CPUs ... [ 0.057593] x86: Booting SMP configuration: [ 0.058015] .... node #0, CPUs: #1 #2 #3 [ 0.062065] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148066] node 0 deferred pages initialised in 81ms [ 0.151121] devtmpfs: initialized [ 0.152297] x86/mm: Memory block size: 128MB [ 0.156104] gcov: version magic: 0x41383552 [ 0.159187] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165357] pinctrl core: initialized pinctrl subsystem [ 0.167239] [ 0.167720] ************************************************************* [ 0.170012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172012] ** ** [ 0.174014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176013] ** ** [ 0.179012] ** This means that this kernel is built to expose internal ** [ 0.181012] ** IOMMU data structures, which may compromise security on ** [ 0.184014] ** your system. ** [ 0.186012] ** ** [ 0.190023] ** If you see this message and you are not debugging the ** [ 0.215035] ** kernel, report this immediately to your vendor! ** [ 0.218018] ** ** [ 0.222021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225065] ************************************************************* [ 0.230260] NET: Registered protocol family 16 [ 0.232516] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.235065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244016] cpuidle: using governor menu [ 0.245791] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.248453] PCI: Using configuration type 1 for base access [ 0.250328] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.260122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.262036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.266180] cryptd: max_cpu_qlen set to 1000 [ 0.269423] ACPI: Added _OSI(Module Device) [ 0.271017] ACPI: Added _OSI(Processor Device) [ 0.273014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.275013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.279736] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.287586] ACPI: Interpreter enabled [ 0.289065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290015] ACPI: Using IOAPIC for interrupt routing [ 0.292131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295287] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.308169] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.310043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.317129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.323526] acpiphp: Slot [2] registered [ 0.325139] acpiphp: Slot [5] registered [ 0.326123] acpiphp: Slot [6] registered [ 0.328204] acpiphp: Slot [7] registered [ 0.330114] acpiphp: Slot [8] registered [ 0.331154] acpiphp: Slot [9] registered [ 0.333128] acpiphp: Slot [10] registered [ 0.335138] acpiphp: Slot [3] registered [ 0.336097] acpiphp: Slot [4] registered [ 0.337139] acpiphp: Slot [11] registered [ 0.338096] acpiphp: Slot [12] registered [ 0.340125] acpiphp: Slot [13] registered [ 0.341159] acpiphp: Slot [14] registered [ 0.343095] acpiphp: Slot [15] registered [ 0.344149] acpiphp: Slot [16] registered [ 0.346100] acpiphp: Slot [17] registered [ 0.348122] acpiphp: Slot [18] registered [ 0.349094] acpiphp: Slot [19] registered [ 0.350067] acpiphp: Slot [20] registered [ 0.352091] acpiphp: Slot [21] registered [ 0.353113] acpiphp: Slot [22] registered [ 0.354079] acpiphp: Slot [23] registered [ 0.355062] acpiphp: Slot [24] registered [ 0.356083] acpiphp: Slot [25] registered [ 0.358107] acpiphp: Slot [26] registered [ 0.359097] acpiphp: Slot [27] registered [ 0.360070] acpiphp: Slot [28] registered [ 0.361059] acpiphp: Slot [29] registered [ 0.363071] acpiphp: Slot [30] registered [ 0.364075] acpiphp: Slot [31] registered [ 0.365064] PCI host bridge to bus 0000:00 [ 0.367018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.371023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.376024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.381181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.382672] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385202] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.393818] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.398908] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.401016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.402011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.405493] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407508] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412745] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.427017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.431016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.438904] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.484300] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539577] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.581017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597483] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.608016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.638042] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653390] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.664017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.691017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703351] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.715076] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.729034] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.761027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.776299] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778464] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.780428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.782372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.785296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790127] iommu: Default domain type: Passthrough [ 0.792276] SCSI subsystem initialized [ 0.794197] ACPI: bus type USB registered [ 0.795140] usbcore: registered new interface driver usbfs [ 0.797126] usbcore: registered new interface driver hub [ 0.799160] usbcore: registered new device driver usb [ 0.800193] pps_core: LinuxPPS API ver. 1 registered [ 0.802014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805080] PTP clock support registered [ 0.807060] EDAC MC: Ver: 3.0.0 [ 0.808615] PCI: Using ACPI for IRQ routing [ 0.810482] NetLabel: Initializing [ 0.812018] NetLabel: domain hash size = 128 [ 0.813009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.815123] NetLabel: unlabeled traffic allowed by default [ 0.817117] vgaarb: loaded [ 0.818303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.820019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.825400] clocksource: Switched to clocksource kvm-clock [ 0.941162] VFS: Disk quotas dquot_6.6.0 [ 0.942854] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.945348] *** VALIDATE ramfs *** [ 0.946087] *** VALIDATE hugetlbfs *** [ 0.947219] pnp: PnP ACPI init [ 0.949336] pnp: PnP ACPI: found 6 devices [ 0.966790] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.970170] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.972306] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974426] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976742] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.979099] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.981967] NET: Registered protocol family 2 [ 0.984330] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.988690] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.992436] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.999075] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.003491] TCP: Hash tables configured (established 65536 bind 65536) [ 1.006927] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.011504] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.014650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.017889] NET: Registered protocol family 1 [ 1.020618] RPC: Registered named UNIX socket transport module. [ 1.022705] RPC: Registered udp transport module. [ 1.024410] RPC: Registered tcp transport module. [ 1.026394] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.029181] NET: Registered protocol family 44 [ 1.031519] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.034895] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.037778] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.040891] PCI: CLS 0 bytes, default 64 [ 1.044183] Unpacking initramfs... [ 2.504277] debug: unmapping init [mem 0xffff99bd3cc54000-0xffff99bd3ffbffff] [ 2.511238] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.513765] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.518414] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.038522] Initialise system trusted keyrings [ 3.040397] Key type blacklist registered [ 3.042473] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052473] zbud: loaded [ 3.055828] *** VALIDATE nfs *** [ 3.057450] *** VALIDATE nfs4 *** [ 3.059335] pstore: using deflate compression [ 3.064026] Platform Keyring initialized [ 3.153048] NET: Registered protocol family 38 [ 3.154862] Key type asymmetric registered [ 3.156011] Asymmetric key parser 'x509' registered [ 3.157600] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.159929] io scheduler mq-deadline registered [ 3.161242] io scheduler kyber registered [ 3.162605] io scheduler bfq registered [ 3.164103] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.166466] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.168896] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.171295] ACPI: Power Button [PWRF] [ 3.175716] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.180969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.190579] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.196605] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.209046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.239890] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.266116] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.271869] Non-volatile memory driver v1.3 [ 3.273826] Linux agpgart interface v0.103 [ 3.305463] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.307658] vda: detected capacity change from 0 to 75096064 [ 3.326546] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.329187] vdb: detected capacity change from 0 to 1073741824 [ 3.347553] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.350355] vdc: detected capacity change from 0 to 2621440000 [ 3.367064] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.370239] vdd: detected capacity change from 0 to 2621440000 [ 3.386844] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.390278] vde: detected capacity change from 0 to 4294967296 [ 3.429974] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.433918] vdf: detected capacity change from 0 to 4294967296 [ 3.443059] libphy: Fixed MDIO Bus: probed [ 3.451956] usbcore: registered new interface driver usbserial_generic [ 3.455284] usbserial: USB Serial support registered for generic [ 3.458265] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.463487] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.465603] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.468337] mousedev: PS/2 mouse device common for all mice [ 3.471755] rtc_cmos 00:05: RTC can wake from S4 [ 3.474776] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.475810] rtc_cmos 00:05: registered as rtc0 [ 3.480608] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.482428] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.484686] intel_pstate: CPU model not supported [ 3.493661] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.499491] hid: raw HID events driver (C) Jiri Kosina [ 3.501477] usbcore: registered new interface driver usbhid [ 3.504572] usbhid: USB HID core driver [ 3.506779] drop_monitor: Initializing network drop monitor service [ 3.510076] Initializing XFRM netlink socket [ 3.512758] NET: Registered protocol family 10 [ 3.517975] Segment Routing with IPv6 [ 3.519658] NET: Registered protocol family 17 [ 3.522680] mpls_gso: MPLS GSO support [ 3.528378] RAS: Correctable Errors collector initialized. [ 3.531270] AVX version of gcm_enc/dec engaged. [ 3.532922] AES CTR mode by8 optimization enabled [ 3.623281] sched_clock: Marking stable (3623253833, 0)->(4522081357, -898827524) [ 3.627828] registered taskstats version 1 [ 3.631090] Loading compiled-in X.509 certificates [ 3.633924] zswap: loaded using pool lzo/zbud [ 3.667893] Key type big_key registered [ 3.684729] Key type encrypted registered [ 3.687024] ima: No TPM chip found, activating TPM-bypass! [ 3.690420] ima: Allocated hash algorithm: sha1 [ 3.693185] ima: No architecture policies found [ 3.695853] evm: Initialising EVM extended attributes: [ 3.697580] evm: security.selinux [ 3.699442] evm: security.ima [ 3.700408] evm: security.capability [ 3.702762] evm: HMAC attrs: 0x1 [ 3.706847] rtc_cmos 00:05: setting system clock to 2026-09-13 02:03:46 UTC (1789265026) [ 3.717598] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.721324] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.727187] Write protecting the kernel read-only data: 28672k [ 3.731190] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.734172] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.781949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.790802] systemd[1]: Detected virtualization kvm. [ 3.792519] systemd[1]: Detected architecture x86-64. [ 3.794467] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.820449] systemd[1]: No hostname configured. [ 3.822377] systemd[1]: Set hostname to . [ 3.824592] random: systemd: uninitialized urandom read (16 bytes read) [ 3.827229] systemd[1]: Initializing machine ID from random generator. [ 3.885849] random: ln: uninitialized urandom read (6 bytes read) [ 4.004158] random: systemd: uninitialized urandom read (16 bytes read) [ 4.007425] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.012915] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.017427] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.721467] device-mapper: uevent: version 1.0.3 [ 4.723893] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.440433] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.508993] virtio_net virtio0 ens2: renamed from eth0 [ 5.624959] scsi host0: ata_piix [ 5.633535] scsi host1: ata_piix [ 5.637265] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.644086] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.337984] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.208128] random: crng init done [ 10.210072] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.696678] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.846086] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.165818] SELinux: Disabled at runtime. [ 12.221402] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.230074] systemd[1]: Detected virtualization kvm. [ 12.232547] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.701470] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.703973] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.708453] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.711522] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.715225] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.721422] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.724921] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File[ 12.862491] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.137431] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.452922] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.465948] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.557185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.570235] EDAC sbridge: Ver: 1.1.2 [ 14.880606] Key type dns_resolver registered [ 15.192680] NFS: Registering the id_resolver key type [ 15.195635] Key type id_resolver registered [ 15.197871] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 68.162769] libcfs: loading out-of-tree module taints kernel. [ 68.265801] Key type ._llcrypt registered [ 68.267854] Key type .llcrypt registered [ 68.361771] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 88.500687] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 89.977982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.993372] alg: No test for adler32 (adler32-zlib) [ 91.614840] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 92.606166] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 94.392220] Key type lgssc registered [ 96.212607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.894455] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.615279] hrtimer: interrupt took 2599557 ns [ 159.023704] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 170.143842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.183787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.494590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.534879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.619922] Lustre: lustre-MDT0000: new disk, initializing [ 171.708126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.743606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.482929] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.145319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.288580] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.327798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.331934] Lustre: Skipped 1 previous similar message [ 190.421946] Lustre: lustre-MDT0001: new disk, initializing [ 190.487816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.543761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.574219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.091798] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.579422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.385727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.752446] Lustre: lustre-OST0000: new disk, initializing [ 210.756661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.763509] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.845857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.099347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 216.109522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 216.138294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 218.157208] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.700912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.826230] Lustre: lustre-OST0001: new disk, initializing [ 233.832798] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.841758] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.908139] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.192895] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.886606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.909420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.981253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.659258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.760940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.820657] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 278.742868] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 284.353978] Lustre: DEBUG MARKER: Client: 2.17.57.105 [ 287.286378] Lustre: DEBUG MARKER: MDS: 2.17.57.105 [ 289.823094] Lustre: DEBUG MARKER: OSS: 2.17.57.105 [ 292.091057] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 22:08:32 EDT 2026 [ 310.602659] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.836385] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.186622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.191422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.212300] Lustre: Skipped 3 previous similar messages [ 329.237094] Lustre: Skipped 3 previous similar messages [ 334.319731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.332819] Lustre: Skipped 3 previous similar messages [ 335.585609] Lustre: server umount lustre-MDT0000 complete [ 339.433398] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.471394] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 340.177432] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265362 with bad export cookie 16879106589424851431 [ 340.197699] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.202929] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 344.552513] LustreError: 9536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.554180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.554612] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.568582] LustreError: 9536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 344.608265] Lustre: Skipped 1 previous similar message [ 345.569158] LustreError: 12161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.587778] LustreError: 12161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 346.721029] Lustre: server umount lustre-MDT0001 complete [ 351.774835] Lustre: server umount lustre-OST0000 complete [ 355.589631] Lustre: server umount lustre-OST0001 complete [ 362.790656] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 373.858866] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 411.300603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.540379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.572276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.659343] Lustre: lustre-MDT0000: new disk, initializing [ 411.733296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.754626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.450813] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 427.660123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.843630] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.945915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.951316] Lustre: Skipped 1 previous similar message [ 428.159914] Lustre: lustre-MDT0001: new disk, initializing [ 428.320290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.403587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.415123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.429913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 446.295776] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.633248] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 459.772484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.061308] Lustre: lustre-OST0000: new disk, initializing [ 460.065511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.072908] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.155286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.183758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.198601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.336702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.836432] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 475.700086] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.564055] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.802782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.362981] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 485.541463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.913264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.917265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.922155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.939278] Lustre: Skipped 1 previous similar message [ 487.968768] Lustre: Skipped 4 previous similar messages [ 493.035217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.037896] Lustre: Skipped 1 previous similar message [ 493.656193] Lustre: server umount lustre-OST0000 complete [ 498.148242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.274026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.284683] Lustre: Skipped 3 previous similar messages [ 504.185718] Lustre: server umount lustre-MDT0000 complete [ 508.385889] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.397346] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 508.940611] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265531 with bad export cookie 16879106589424852964 [ 508.950620] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.275534] Lustre: server umount lustre-MDT0001 complete [ 516.035523] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:12:16 (1789265536) [ 534.353538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.665347] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.754774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.703745] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 544.247727] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.266540] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 548.548125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.318291] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 562.047605] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 569.324300] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.445947] Lustre: Failing over lustre-MDT0000 [ 572.668352] Lustre: server umount lustre-MDT0000 complete [ 573.484807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.511303] Lustre: Skipped 1 previous similar message [ 573.519308] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.540397] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 589.611480] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265596/real 1789265596] req@ffff99bc83188e00 x1876180457388288/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789265612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 589.649715] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.342353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.922130] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xea3ea203849fa56a [ 599.939416] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 600.444496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.460033] Lustre: Skipped 1 previous similar message [ 604.366419] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 605.672280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 605.682477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 605.721728] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 616.143892] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 621.026973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.047292] LustreError: Skipped 1 previous similar message [ 621.063654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.080819] Lustre: Skipped 1 previous similar message [ 621.095319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.968527] Lustre: server umount lustre-MDT0000 complete [ 629.217235] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.237106] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 631.496443] LustreError: 20662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265654 with bad export cookie 16879106589424854378 [ 631.498270] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.514471] LustreError: 20662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.215106] Lustre: server umount lustre-MDT0001 complete [ 641.100609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.370560] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 656.438824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.597526] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 673.298544] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 679.013516] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 681.532532] Lustre: Failing over lustre-MDT0000 [ 681.788734] Lustre: server umount lustre-MDT0000 complete [ 682.472598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.499564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.537567] Lustre: Skipped 1 previous similar message [ 696.801893] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.826059] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 702.029336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.144370] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.500852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 707.512367] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 707.577787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 707.588597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 707.591188] Lustre: Skipped 1 previous similar message [ 707.618092] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 716.743732] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 722.922686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 722.931940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.966686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.980198] Lustre: Skipped 2 previous similar messages [ 722.997752] Lustre: Skipped 3 previous similar messages [ 727.128134] Lustre: server umount lustre-MDT0000 complete [ 731.441595] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265754 with bad export cookie 16879106589424855316 [ 731.451536] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.882416] Lustre: server umount lustre-MDT0001 complete [ 740.899482] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:16:01 (1789265761) [ 759.570822] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 771.591815] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 818.494527] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 828.491864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.718624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 828.738691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 828.788909] Lustre: lustre-MDT0000: new disk, initializing [ 828.851794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.860230] Lustre: Skipped 3 previous similar messages [ 828.873711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.094931] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 842.916822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.035673] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 843.280091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 848.360608] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 857.466687] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 862.683845] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 869.892902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.124350] Lustre: 31612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 871.287330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 871.339420] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 871.369689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 871.387982] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 876.222771] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 884.478528] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.027340] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 907.234251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 907.246453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.249215] LustreError: Skipped 1 previous similar message [ 907.264392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.280666] Lustre: Skipped 1 previous similar message [ 907.302287] Lustre: Skipped 1 previous similar message [ 911.963224] Lustre: server umount lustre-OST0000 complete [ 921.976163] Lustre: server umount lustre-MDT0000 complete [ 922.594250] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.630199] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 925.716741] LustreError: 32840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265948 with bad export cookie 16879106589424855722 [ 925.721046] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.728684] LustreError: 32840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 926.200960] Lustre: server umount lustre-MDT0001 complete [ 937.744784] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 941.003390] Key type lgssc unregistered [ 941.451722] LNet: 33590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.458200] LNetError: 33590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.483119] LNet: Removed LNI 192.168.202.149@tcp [ 942.516367] Key type .llcrypt unregistered [ 942.518806] Key type ._llcrypt unregistered [ 969.055727] Key type ._llcrypt registered [ 969.061925] Key type .llcrypt registered [ 969.178568] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 970.416106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.432760] alg: No test for adler32 (adler32-zlib) [ 971.660386] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 971.852326] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 973.498935] Key type lgssc registered [ 974.684958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 983.751067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 983.767915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.235269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.450221] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 994.792665] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.803721] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 999.923770] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.240556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.751385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.917940] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1014.690402] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.916635] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1028.153660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1028.673539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.521681] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1042.839254] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1047.097891] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1060.055879] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:21:20 (1789266080) [ 1061.859738] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1063.516600] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1069.529085] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:21:30 (1789266090) [ 1077.511461] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:21:37 (1789266097) [ 1078.775376] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1088.235197] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1088.240136] Lustre: Skipped 2 previous similar messages [ 1089.963528] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:21:50 (1789266110) [ 1093.597241] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1093.602452] Lustre: Skipped 3 previous similar messages [ 1110.076659] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1110.081344] Lustre: Skipped 6 previous similar messages [ 1112.289936] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:22:13 (1789266133) [ 1115.000626] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1118.665448] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1126.762257] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:22:27 (1789266147) [ 1128.491248] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1128.495432] Lustre: Skipped 3 previous similar messages [ 1142.231945] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:22:43 (1789266163) [ 1157.601386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.609062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.623111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.652747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.653951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.672779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.840583] Lustre: server umount lustre-OST0000 complete [ 1169.889609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.908314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.916083] Lustre: Skipped 1 previous similar message [ 1169.935037] Lustre: Skipped 2 previous similar messages [ 1172.943885] Lustre: server umount lustre-MDT0000 complete [ 1175.016034] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.043612] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1177.072262] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266199 with bad export cookie 14732218360087894867 [ 1177.085250] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.410553] Lustre: server umount lustre-MDT0001 complete [ 1189.111928] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1192.022968] Key type lgssc unregistered [ 1192.391946] LNet: 42840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.400213] LNetError: 42840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.432531] LNet: Removed LNI 192.168.202.149@tcp [ 1193.484852] Key type .llcrypt unregistered [ 1193.486670] Key type ._llcrypt unregistered [ 1211.924243] Key type ._llcrypt registered [ 1211.926511] Key type .llcrypt registered [ 1212.011660] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 1227.428168] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1228.335833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.383945] alg: No test for adler32 (adler32-zlib) [ 1229.556710] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1229.801895] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1231.520206] Key type lgssc registered [ 1233.284902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.954144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.997988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.175202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1271.194815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1271.258377] Lustre: lustre-MDT0000: new disk, initializing [ 1271.320369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.337238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1276.184833] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1286.283511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.406408] Lustre: 45994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1286.447420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1286.451645] Lustre: Skipped 1 previous similar message [ 1286.515712] Lustre: lustre-MDT0001: new disk, initializing [ 1286.618283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.685222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1286.707846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1290.857510] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1299.102528] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.642384] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1312.792700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.186845] Lustre: lustre-OST0000: new disk, initializing [ 1313.195886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1313.204463] Lustre: 47961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1313.321485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.409276] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1322.552391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1322.556959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1322.658247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1327.854752] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1332.396181] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1332.577361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1336.910659] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1337.099867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1343.459161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1343.470862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.474941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1343.484550] Lustre: Skipped 1 previous similar message [ 1348.586772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1348.590967] Lustre: Skipped 2 previous similar messages [ 1353.184181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1353.444697] Lustre: server umount lustre-OST0000 complete [ 1358.305737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.314264] LustreError: Skipped 1 previous similar message [ 1358.320182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.340349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1363.050863] Lustre: server umount lustre-MDT0000 complete [ 1366.443283] LustreError: 45987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266389 with bad export cookie 4787702011844215167 [ 1366.445221] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.455062] LustreError: 45987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1366.734623] Lustre: server umount lustre-MDT0001 complete [ 1376.261661] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:26:36 (1789266396) [ 1382.158956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.458409] LustreError: 50422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.479088] LustreError: 50422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1382.511076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.504649] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1388.022497] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.034236] LustreError: 50422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.243135] LustreError: 50422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.263533] LustreError: 50422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1398.851550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.127058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.529169] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1411.435559] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1417.434192] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1424.825244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.082048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.331734] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1438.984463] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1448.027460] LustreError: 52950:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1448.855359] Lustre: Modifying parameter general.timeout=1 in log params [ 1449.995733] Lustre: Modifying parameter general.timeout=2 in log params [ 1452.481812] Lustre: Modifying parameter general.timeout=4 in log params [ 1452.488169] Lustre: Skipped 1 previous similar message [ 1456.986359] Lustre: Modifying parameter general.timeout=8 in log params [ 1456.993111] Lustre: Skipped 3 previous similar messages [ 1465.770979] Lustre: Modifying parameter general.timeout=16 in log params [ 1465.773630] Lustre: Skipped 7 previous similar messages [ 1482.544109] Lustre: Modifying parameter general.timeout=33 in log params [ 1482.550357] Lustre: Skipped 16 previous similar messages [ 1515.695644] Lustre: Modifying parameter general.timeout=64 in log params [ 1515.702268] Lustre: Skipped 30 previous similar messages [ 1580.202314] Lustre: Modifying parameter general.timeout=127 in log params [ 1580.208778] Lustre: Skipped 62 previous similar messages [ 1594.293611] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:30:14 (1789266614) [ 1597.522951] Lustre: 59902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1599.559765] Lustre: 60000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1603.117077] Lustre: 60214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1603.123622] Lustre: 60214:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1632.904574] Lustre: 61773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1632.915592] Lustre: 61773:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1632.939490] LustreError: 61773:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1637.566478] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:30:57 (1789266657) [ 1653.731636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.738546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.764988] Lustre: Skipped 3 previous similar messages [ 1653.786715] Lustre: Skipped 3 previous similar messages [ 1656.670710] Lustre: server umount lustre-MDT0000 complete [ 1658.850028] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1658.882784] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1660.151622] LustreError: 57463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266682 with bad export cookie 4787702011844216049 [ 1660.160393] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.162607] LustreError: 57463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1660.475922] Lustre: server umount lustre-MDT0001 complete [ 1674.220294] Lustre: server umount lustre-OST0000 complete [ 1705.262287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.609729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1706.071989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.864231] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1721.788873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.932093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1721.985821] Lustre: 64309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1727.004429] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1735.704528] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1742.565514] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1749.949074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.242946] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1750.482925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.486709] Lustre: Skipped 1 previous similar message [ 1757.556381] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1766.647215] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1768.448794] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1768.457972] Lustre: Skipped 26 previous similar messages [ 1768.520739] Lustre: 66873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1776.609333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.616989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.631042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.679369] Lustre: Skipped 1 previous similar message [ 1782.262178] Lustre: server umount lustre-OST0000 complete [ 1786.849463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.851695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.874414] Lustre: Skipped 1 previous similar message [ 1786.898847] Lustre: Skipped 4 previous similar messages [ 1792.871320] Lustre: server umount lustre-MDT0000 complete [ 1797.089508] LustreError: 64315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.123685] LustreError: 64315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1797.491702] LustreError: 64299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266820 with bad export cookie 4787702011844219178 [ 1797.516637] LustreError: 64299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1797.516851] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.202659] Lustre: server umount lustre-MDT0001 complete [ 1812.705101] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1815.599343] Key type lgssc unregistered [ 1815.974871] LNet: 68082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.982216] LNetError: 68082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.000658] LNet: Removed LNI 192.168.202.149@tcp [ 1818.033346] Key type .llcrypt unregistered [ 1818.035401] Key type ._llcrypt unregistered [ 1849.178833] Key type ._llcrypt registered [ 1849.181270] Key type .llcrypt registered [ 1849.300113] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1850.121815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.306095] alg: No test for adler32 (adler32-zlib) [ 1851.440409] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1851.622906] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1853.257303] Key type lgssc registered [ 1854.554645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.178391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.201198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.985035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.852764] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1874.412082] LustreError: 69493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.458232] LustreError: 69493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1879.527537] LustreError: 69494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.218826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.048659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.339775] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1895.096781] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.306850] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1908.258993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.946066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.955652] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1923.939665] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.713106] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1940.966716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.972130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.009658] Lustre: Skipped 1 previous similar message [ 1941.036147] Lustre: Skipped 2 previous similar messages [ 1946.103727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.115599] Lustre: Skipped 2 previous similar messages [ 1946.432520] Lustre: server umount lustre-MDT0000 complete [ 1950.031590] LustreError: 69475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266972 with bad export cookie 907055951860932634 [ 1950.032781] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.046616] LustreError: 69475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1950.436172] Lustre: server umount lustre-MDT0001 complete [ 1963.658967] Lustre: server umount lustre-OST0000 complete [ 1988.498522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.754161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1988.790549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1989.047994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.307546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2001.955971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.195338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2002.249699] Lustre: 74362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2002.816222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.222089] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2014.427775] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2020.932388] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2028.066361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.274419] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2028.829705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.677673] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2044.809574] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2046.552561] Lustre: 76939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2048.346542] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2055.652931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.658723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.681770] Lustre: Skipped 1 previous similar message [ 2055.702898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.162270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.817297] Lustre: server umount lustre-OST0000 complete [ 2063.851671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2063.866683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.888442] Lustre: Skipped 1 previous similar message [ 2063.896409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.916553] Lustre: Skipped 2 previous similar messages [ 2066.403642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.592972] Lustre: server umount lustre-MDT0000 complete [ 2071.521088] LustreError: 74375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.544689] LustreError: 74375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2073.157205] LustreError: 74357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267095 with bad export cookie 907055951860933481 [ 2073.160535] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.166506] LustreError: 74357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2073.522787] Lustre: server umount lustre-MDT0001 complete [ 2084.246781] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2087.102626] Key type lgssc unregistered [ 2087.476460] LNet: 78147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.481620] LNetError: 78147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.523111] LNet: Removed LNI 192.168.202.149@tcp [ 2088.332252] Key type .llcrypt unregistered [ 2088.334345] Key type ._llcrypt unregistered [ 2115.661077] Key type ._llcrypt registered [ 2115.676412] Key type .llcrypt registered [ 2115.822786] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2117.089455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.101443] alg: No test for adler32 (adler32-zlib) [ 2118.499626] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2118.995694] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2120.816360] Key type lgssc registered [ 2122.880828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2133.041632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2133.065213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.454868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.856853] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2144.741440] LustreError: 79558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.765948] LustreError: 79558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2148.698161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.550529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.497481] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2161.495598] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.434902] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2173.929824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.469625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.479131] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2187.710605] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2196.392964] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2197.289920] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.077175] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.083434] Lustre: Skipped 1 previous similar message [ 2201.830641] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2201.836645] Lustre: Skipped 2 previous similar messages [ 2206.057971] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2206.062555] Lustre: Skipped 4 previous similar messages [ 2214.607980] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2214.613600] Lustre: Skipped 8 previous similar messages [ 2231.138150] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2231.145602] Lustre: Skipped 17 previous similar messages [ 2263.372234] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.383695] Lustre: Skipped 30 previous similar messages [ 2307.005346] Lustre: 87662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2309.832627] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:42:10 (1789267330) [ 2311.247671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2313.841480] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:42:13 (1789267333) [ 2321.892427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.907720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2321.911483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2324.414083] Lustre: server umount lustre-OST0000 complete [ 2328.546978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.553974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.564513] Lustre: Skipped 1 previous similar message [ 2328.581519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.589531] Lustre: Skipped 1 previous similar message [ 2332.130027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.131788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.158308] Lustre: Skipped 1 previous similar message [ 2334.559993] Lustre: server umount lustre-MDT0000 complete [ 2337.250004] LustreError: 82060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.264391] LustreError: 82060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2338.428938] LustreError: 84126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267361 with bad export cookie 1397539648796541615 [ 2338.429376] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.442571] LustreError: 84126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2338.774851] Lustre: server umount lustre-MDT0001 complete [ 2350.793593] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2354.135383] Key type lgssc unregistered [ 2354.380576] LNet: 88910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.387940] LNetError: 88910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.405776] LNet: Removed LNI 192.168.202.149@tcp [ 2355.408591] Key type .llcrypt unregistered [ 2355.413789] Key type ._llcrypt unregistered [ 2360.600400] Key type ._llcrypt registered [ 2360.601976] Key type .llcrypt registered [ 2360.686366] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2366.238154] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules [ 2368.001984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.042514] alg: No test for adler32 (adler32-zlib) [ 2369.070891] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2369.331937] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2371.008189] Key type lgssc registered [ 2372.666697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2374.220589] LustreError: 90118:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2393.171740] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2395.586635] LustreError: 90118:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2401.375397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.392055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.744996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.744672] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2413.031187] LustreError: 91106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.058302] LustreError: 91106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2415.053296] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:43:55 (1789267435) [ 2418.154858] LustreError: 91107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.659652] LustreError: 91138:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2419.674609] LustreError: 91138:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2419.682019] LustreError: 91138:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2420.093535] Lustre: server umount lustre-MDT0000 complete [ 2434.031487] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2436.485765] Key type lgssc unregistered [ 2436.750968] LNet: 92630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.755486] LNetError: 92630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.776906] LNet: Removed LNI 192.168.202.149@tcp [ 2437.402598] Key type .llcrypt unregistered [ 2437.412814] Key type ._llcrypt unregistered [ 2454.995665] Key type ._llcrypt registered [ 2454.999687] Key type .llcrypt registered [ 2455.167233] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2456.387406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.540363] alg: No test for adler32 (adler32-zlib) [ 2457.530978] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2457.733412] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2459.377541] Key type lgssc registered [ 2460.616340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2469.386040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2469.421512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.908938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.443481] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2481.129144] LustreError: 93846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.150390] LustreError: 93846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2485.343418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.144485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.812860] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2498.060758] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2503.420745] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2509.939626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.474710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.576925] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2523.422807] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2571.809687] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:46:31 (1789267591) [ 2573.922583] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2576.102540] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:46:36 (1789267596) [ 2582.497845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.504665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.526276] Lustre: Skipped 2 previous similar messages [ 2582.554684] Lustre: Skipped 2 previous similar messages [ 2587.632667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.637656] Lustre: Skipped 2 previous similar messages [ 2588.548869] Lustre: server umount lustre-MDT0000 complete [ 2592.738115] LustreError: 93846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.762228] LustreError: 93846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2592.979951] LustreError: 94586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267615 with bad export cookie 995915906051764173 [ 2592.988137] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2592.996756] LustreError: 94586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2593.303560] Lustre: server umount lustre-MDT0001 complete [ 2608.342368] Lustre: server umount lustre-OST0000 complete [ 2622.943223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.625566] LustreError: 98122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.960582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.098564] LustreError: 98123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.582037] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2634.215632] LustreError: 98122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.013591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.711725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.091883] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2649.749712] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2655.890895] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2669.814386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.013086] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2670.020876] LustreError: 98113:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2670.064770] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2670.077879] LustreError: 100206:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2670.088928] LustreError: 100206:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2670.097561] LustreError: 100206:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2670.164446] Lustre: server umount lustre-OST0000 complete [ 2670.166666] LustreError: 100206:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2679.890100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.096443] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2680.105100] LustreError: 98110:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2680.137453] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2680.148879] LustreError: 100672:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2680.156880] LustreError: 100672:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2680.164607] LustreError: 100672:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2680.231926] Lustre: server umount lustre-OST0000 complete [ 2680.235833] LustreError: 100672:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2690.338364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.478070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2690.482864] Lustre: Found index 0 for lustre-OST0000, updating log [ 2690.488458] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2690.549754] Lustre: lustre-OST0000: new disk, initializing [ 2690.552906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2690.557959] Lustre: 101206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2690.904034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.821664] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2701.657295] Lustre: Failing over lustre-OST0000 [ 2701.827592] Lustre: server umount lustre-OST0000 complete [ 2707.818347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.835160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.846082] Lustre: Skipped 1 previous similar message [ 2713.063119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.070136] Lustre: Skipped 2 previous similar messages [ 2718.187146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.192410] Lustre: Skipped 1 previous similar message [ 2720.224282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2720.562066] Lustre: server umount lustre-MDT0000 complete [ 2723.297890] LustreError: 98117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.321033] LustreError: 98117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2726.108188] LustreError: 98103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267748 with bad export cookie 995915906051766917 [ 2726.113705] LustreError: 98103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2726.114063] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.592176] Lustre: server umount lustre-MDT0001 complete [ 2735.684478] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:49:15 (1789267755) [ 2753.145828] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2762.288646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.914179] LustreError: 103553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.116358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.755698] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2777.486870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.555733] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2788.215828] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2795.784375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.833384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.836554] Lustre: Skipped 1 previous similar message [ 2796.862712] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2802.981700] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2802.999501] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2803.020404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2806.768468] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2808.231629] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2808.252323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2808.269639] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2808.278225] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2808.319260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2817.508657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.674783] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2817.839474] Lustre: lustre-OST0001: new disk, initializing [ 2817.845792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2817.859588] Lustre: 105874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2823.844602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2823.857303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2823.972901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2826.443415] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2834.501637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2840.438280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2845.160344] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2845.171907] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.184576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2849.771691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.255518] Lustre: server umount lustre-MDT0001 complete [ 2854.889155] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.922826] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2860.567592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.719529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2860.724811] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2860.782028] Lustre: 103542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2861.726309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2861.737708] Lustre: Skipped 1 previous similar message [ 2862.365909] LustreError: 107408:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2862.377315] LustreError: 107408:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2862.393786] LustreError: 103545:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2862.524056] LustreError: 107418:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.149@tcp: cfg command failed: rc = -17 [ 2862.547788] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2862.547788] [ 2866.649329] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2867.187374] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2875.571287] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:51:36 (1789267896) [ 2893.382457] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2911.668732] Lustre: 109485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2934.318721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2937.600773] Lustre: 110622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3014.628244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.641133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.658475] Lustre: Skipped 1 previous similar message [ 3014.664531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.667250] Lustre: Skipped 4 previous similar messages [ 3020.592536] Lustre: server umount lustre-MDT0000 complete [ 3020.773980] LustreError: 105274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.789836] LustreError: 105274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3024.738644] LustreError: 103535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268047 with bad export cookie 995915906051767918 [ 3024.746614] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.760481] LustreError: 103535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3025.006779] Lustre: server umount lustre-MDT0001 complete [ 3039.497208] Lustre: server umount lustre-OST0000 complete [ 3041.251193] Lustre: 93157:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789268048/real 1789268048] req@ffff99bc8eea5880 x1876182937464192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789268064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3041.283497] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.297222] Lustre: Skipped 3 previous similar messages [ 3044.159351] Lustre: server umount lustre-OST0001 complete [ 3051.736665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3062.302680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3070.642657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3080.346910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3099.104881] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3110.358263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.382287] Lustre: lustre-MDT0000: reset Object Index mappings [ 3110.787164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.079354] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3123.591732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.633908] Lustre: lustre-MDT0001: reset Object Index mappings [ 3129.123775] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3131.737472] Lustre: 115740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3138.671586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.453287] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3148.854284] LustreError: 116095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.879774] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3148.884439] LustreError: 116095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3154.479215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.091182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3156.098142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3156.144291] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3160.711408] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3168.952609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3172.483466] Lustre: 117617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3218.579810] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:57:18 (1789268238) [ 3247.078473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.092820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.114567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.122307] Lustre: Skipped 4 previous similar messages [ 3259.360174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3259.736447] Lustre: server umount lustre-MDT0000 complete [ 3264.666737] LustreError: 114571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268287 with bad export cookie 995915906051814986 [ 3264.675387] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.692550] LustreError: 114571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3271.317918] Lustre: server umount lustre-MDT0001 complete [ 3282.207776] Lustre: server umount lustre-OST0000 complete [ 3301.344688] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.573814] Lustre: server umount lustre-OST0001 complete [ 3310.167591] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3319.803796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3356.102660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.289951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3356.318993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3356.378571] Lustre: lustre-MDT0000: new disk, initializing [ 3356.444573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3361.113845] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3368.710883] Lustre: server umount lustre-MDT0000 complete [ 3379.551481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.828069] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3379.843951] Lustre: 121883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3379.860693] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3380.072620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.077581] Lustre: Skipped 4 previous similar messages [ 3385.729255] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3391.586832] Lustre: server umount lustre-MDT0000 complete [ 3402.163178] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:00:22 (1789268422) [ 3403.859810] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3406.212664] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:00:26 (1789268426) [ 3424.204327] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3434.430799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.836495] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3448.228153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.346873] Lustre: 123938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3448.390111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3448.393447] Lustre: Skipped 1 previous similar message [ 3448.448578] Lustre: lustre-MDT0001: new disk, initializing [ 3448.660272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3448.675979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3453.087220] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3457.613934] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3465.299940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3465.548379] Lustre: lustre-OST0000: new disk, initializing [ 3465.552419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3465.559429] Lustre: 125529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3466.904115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3466.925803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3467.038231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3472.146518] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3482.720479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.885417] Lustre: lustre-OST0001: new disk, initializing [ 3482.889501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3482.895491] Lustre: 126400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3484.740936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3484.755327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3484.869510] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3489.573769] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3498.286766] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3501.647412] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3889.874224] Lustre: 132816:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3889.884959] Lustre: 132816:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3889.894215] Lustre: 132816:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3889.902787] Lustre: 132816:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3889.909283] Lustre: 132816:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3889.917850] Lustre: 132816:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4143.316974] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:12:43 (1789269163) [ 4160.993639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4161.001212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.004042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4161.004049] Lustre: Skipped 14 previous similar messages [ 4161.034781] Lustre: Skipped 6 previous similar messages [ 4164.563261] Lustre: server umount lustre-MDT0000 complete [ 4166.117914] LustreError: 126494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.138992] LustreError: 126494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4167.975743] LustreError: 123931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269190 with bad export cookie 995915906051881318 [ 4167.984476] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4167.991155] LustreError: 123931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4171.238682] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.242184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4171.248342] Lustre: Skipped 3 previous similar messages [ 4171.261526] Lustre: Skipped 4 previous similar messages [ 4174.725807] Lustre: server umount lustre-MDT0001 complete [ 4193.251021] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4193.458213] Lustre: server umount lustre-OST0000 complete [ 4203.235528] Lustre: server umount lustre-OST0001 complete [ 4209.474534] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4218.813957] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4254.514670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.758416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4254.797170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4254.904853] Lustre: lustre-MDT0000: new disk, initializing [ 4255.015784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.022500] Lustre: Skipped 4 previous similar messages [ 4255.036715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4259.503404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4269.056096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.251723] Lustre: 135863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4269.334428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4269.337494] Lustre: Skipped 1 previous similar message [ 4269.465123] Lustre: lustre-MDT0001: new disk, initializing [ 4269.687301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4269.698939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.969828] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4283.295552] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4288.205216] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4295.654834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4295.972333] Lustre: lustre-OST0000: new disk, initializing [ 4295.976206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4295.984767] Lustre: 137828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4297.910423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4297.923140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4298.033804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4302.390592] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4313.274708] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4324.654461] Lustre: lustre-MDD0000: changelog on [ 4328.049738] Lustre: lustre-MDD0001: changelog on [ 5890.814219] Lustre: lustre-MDD0001: changelog off [ 5892.738097] Lustre: 136739:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5893.541867] Lustre: lustre-MDD0000: changelog off [ 5900.771690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5900.773978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.784281] LustreError: Skipped 1 previous similar message [ 5900.785492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5900.785509] Lustre: Skipped 2 previous similar messages [ 5900.833946] Lustre: Skipped 1 previous similar message [ 5905.895834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.912147] Lustre: Skipped 2 previous similar messages [ 5911.017223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.025939] Lustre: Skipped 1 previous similar message [ 5914.081480] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5914.308711] Lustre: server umount lustre-OST0000 complete [ 5921.252105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.274954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.367744] Lustre: server umount lustre-MDT0000 complete [ 5926.381324] LustreError: 136739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.402978] LustreError: 136739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5929.696331] LustreError: 135856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270952 with bad export cookie 995915906052701543 [ 5929.726091] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5929.727097] LustreError: 135856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5930.068716] Lustre: server umount lustre-MDT0001 complete [ 5941.215852] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 5943.542923] Key type lgssc unregistered [ 5943.823059] LNet: 146910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5943.826849] LNetError: 146910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5943.840141] LNet: Removed LNI 192.168.202.149@tcp [ 5944.848150] Key type .llcrypt unregistered [ 5944.852262] Key type ._llcrypt unregistered [ 5966.206391] Key type ._llcrypt registered [ 5966.209935] Key type .llcrypt registered [ 5966.345456] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 5981.183353] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 5982.457632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5982.514333] alg: No test for adler32 (adler32-zlib) [ 5983.523304] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 5983.731654] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 5985.448156] Key type lgssc registered [ 5986.318868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6023.822846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6023.885572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.301663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6025.342718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6025.445618] Lustre: lustre-MDT0000: new disk, initializing [ 6025.533263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.561213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6030.997812] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6041.546979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.716531] Lustre: 150109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6041.804098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6041.816842] Lustre: Skipped 1 previous similar message [ 6041.931626] Lustre: lustre-MDT0001: new disk, initializing [ 6042.080860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6042.147958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6042.173542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6047.290660] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6056.820858] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6062.913640] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.981235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.349973] Lustre: lustre-OST0000: new disk, initializing [ 6070.354253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6070.365057] Lustre: 152081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6070.500238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.575681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6077.599556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6077.619927] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6077.798129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6088.091579] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6094.226846] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module obdecho/obdecho [ 6095.826911] LustreError: 153083:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6095.835615] LustreError: 153083:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6097.314456] LustreError: 153180:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6108.646025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.649362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.659163] Lustre: Skipped 2 previous similar messages [ 6108.660916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6108.687203] Lustre: Skipped 2 previous similar messages [ 6113.764935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.770245] Lustre: Skipped 2 previous similar messages [ 6114.128521] Lustre: server umount lustre-MDT0000 complete [ 6117.944485] LustreError: 150101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271140 with bad export cookie 4937643656539333825 [ 6117.951183] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.953980] LustreError: 150101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6118.296597] Lustre: server umount lustre-MDT0001 complete [ 6133.350623] Lustre: server umount lustre-OST0000 complete [ 6153.422099] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6163.839577] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6197.466781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.658733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6197.688707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6197.754743] Lustre: lustre-MDT0000: new disk, initializing [ 6197.817891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.844219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6202.372908] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6213.777713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.871092] Lustre: 156895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6213.952554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6213.961062] Lustre: Skipped 1 previous similar message [ 6213.963606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6214.050243] Lustre: lustre-MDT0001: new disk, initializing [ 6214.144578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6214.196345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6214.210530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6218.642063] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6228.662717] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6234.228955] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6241.409742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.744754] Lustre: lustre-OST0000: new disk, initializing [ 6241.747899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6241.755949] Lustre: 158862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6241.838803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.076515] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6249.598925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6249.613134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6249.768955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6257.641812] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.128546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6263.300244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6268.155440] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6268.443347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6270.436222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.456932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6270.462364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.472088] Lustre: Skipped 1 previous similar message [ 6275.561064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.573764] Lustre: Skipped 1 previous similar message [ 6276.554565] Lustre: server umount lustre-OST0000 complete [ 6280.673699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.700505] Lustre: Skipped 1 previous similar message [ 6285.797852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.811121] Lustre: Skipped 3 previous similar messages [ 6286.559469] Lustre: server umount lustre-MDT0000 complete [ 6290.591629] LustreError: 156887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271313 with bad export cookie 4937643656539334777 [ 6290.609865] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6290.982529] Lustre: server umount lustre-MDT0001 complete [ 6294.774053] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:48:35 (1789271315) [ 6297.129375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6299.329163] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:48:39 (1789271319) [ 6307.145274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.738714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.358493] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6312.930625] LustreError: 161275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.954571] LustreError: 161275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6318.058370] LustreError: 161274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.182762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.592876] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6333.830566] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6339.042253] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6345.398367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.820634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.832617] Lustre: Skipped 1 previous similar message [ 6351.292372] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6359.833643] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6365.862732] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 6367.497889] Lustre: server umount lustre-MDT0000 complete [ 6367.715062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6367.719277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.727689] Lustre: Skipped 1 previous similar message [ 6367.740211] LustreError: 161274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.939542] LustreError: 161271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.963053] LustreError: 161271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6371.472643] LustreError: 161255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271394 with bad export cookie 4937643656539335666 [ 6371.476027] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6371.492931] LustreError: 161255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6373.858048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6373.892531] Lustre: Skipped 2 previous similar messages [ 6376.055411] LustreError: 161269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.079220] LustreError: 161269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6377.923612] Lustre: server umount lustre-MDT0001 complete [ 6384.535441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6400.734715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.979311] LustreError: 165118:0:(llog.c:1655:llog_backup()) MGC192.168.202.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6400.993280] Lustre: 165118:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6401.516487] LustreError: 165123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.546946] LustreError: 165123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6418.039767] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.054384] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6452.207402] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.228649] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6461.922539] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6461.928725] Lustre: 165118:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6502.933304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.722439] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6508.007732] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6515.929378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.152798] LustreError: 165862:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6517.162595] LustreError: 165862:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000002d8b672a x1876186634508416/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789271539 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6517.197343] LustreError: 165862:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6517.226029] LustreError: 163106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.245914] LustreError: 163106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 6521.459887] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6522.868775] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6528.801100] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6530.560520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6537.060715] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6538.600551] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6558.691937] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.697199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6558.707302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6558.727574] Lustre: Skipped 5 previous similar messages [ 6564.342652] Lustre: server umount lustre-MDT0000 complete [ 6568.514305] LustreError: 163119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271591 with bad export cookie 4937643656539342120 [ 6568.522535] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.536031] LustreError: 163119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6569.178455] Lustre: server umount lustre-MDT0001 complete [ 6575.529392] Lustre: server umount lustre-OST0000 complete [ 6599.798695] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6612.373404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6651.109185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.513732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6651.629890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6651.827973] Lustre: lustre-MDT0000: new disk, initializing [ 6651.973141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.982706] Lustre: Skipped 1 previous similar message [ 6652.016421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6656.480708] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6666.134914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.293257] Lustre: 170226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6666.606251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6671.106063] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6681.590536] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6687.746182] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6696.849568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.228590] Lustre: lustre-OST0000: new disk, initializing [ 6697.232648] Lustre: Skipped 1 previous similar message [ 6697.240462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6697.245833] Lustre: Skipped 2 previous similar messages [ 6697.254459] Lustre: 172195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6706.205330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6706.215646] Lustre: Skipped 1 previous similar message [ 6706.222490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6706.314045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6707.174420] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6717.267370] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.513548] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6723.721235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6729.681928] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6729.928392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6732.258742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.272681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.292956] Lustre: Skipped 3 previous similar messages [ 6732.299295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.302459] Lustre: Skipped 3 previous similar messages [ 6738.415308] Lustre: server umount lustre-OST0000 complete [ 6748.096756] Lustre: server umount lustre-MDT0000 complete [ 6751.831732] LustreError: 170218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271774 with bad export cookie 4937643656539358955 [ 6751.834460] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6751.839383] LustreError: 170218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6752.281343] Lustre: server umount lustre-MDT0001 complete [ 6760.667846] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:56:21 (1789271781) [ 6767.270389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.771511] LustreError: 174605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.794976] LustreError: 174605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6772.044885] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6779.570900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.000724] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6791.021409] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.655154] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6801.914962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.897377] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6814.993706] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.015149] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:57:25 (1789271845) [ 6830.050054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6830.064826] LustreError: Skipped 1 previous similar message [ 6830.071887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.086114] Lustre: Skipped 3 previous similar messages [ 6835.347731] Lustre: server umount lustre-OST0000 complete [ 6845.591787] Lustre: server umount lustre-MDT0000 complete [ 6849.136643] LustreError: 174586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271871 with bad export cookie 4937643656539359837 [ 6849.141991] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.146576] LustreError: 174586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6849.401995] Lustre: server umount lustre-MDT0001 complete [ 6860.466478] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 6863.212432] Key type lgssc unregistered [ 6863.482440] LNet: 178331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6863.488628] LNetError: 178331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6863.506249] LNet: Removed LNI 192.168.202.149@tcp [ 6864.591889] Key type .llcrypt unregistered [ 6864.602680] Key type ._llcrypt unregistered [ 6883.425357] Key type ._llcrypt registered [ 6883.427930] Key type .llcrypt registered [ 6883.561409] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6884.808881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6884.943385] alg: No test for adler32 (adler32-zlib) [ 6885.986426] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 6886.247404] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 6887.928543] Key type lgssc registered [ 6889.247940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6897.392524] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6897.432314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.992577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6918.112455] LustreError: 179571:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.149@tcp: failed processing log, type 4: rc = -110 [ 6923.233438] LustreError: 179571:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6923.251987] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6923.256817] LustreError: MGC192.168.202.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6923.273847] LustreError: 179571:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6923.289489] LustreError: 179571:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6923.298486] LustreError: 179571:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6923.358254] Lustre: server umount lustre-OST0000 complete [ 6923.360220] LustreError: 179571:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6930.464561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.802569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.273985] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6942.191229] LustreError: 180071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.223113] LustreError: 180071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6944.805981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.187594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.614463] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6956.975815] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6962.014561] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6969.899447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.125586] LustreError: 181888:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6970.134286] Lustre: 181888:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6970.253161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.282600] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6985.391223] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.476491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.483130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.499514] Lustre: Skipped 1 previous similar message [ 6989.529869] Lustre: Skipped 2 previous similar messages [ 6993.747026] Lustre: server umount lustre-MDT0000 complete [ 6994.920022] LustreError: 180067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.938441] LustreError: 180067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6997.474808] LustreError: 180053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272020 with bad export cookie 10923538066853572009 [ 6997.491842] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.495898] LustreError: 180053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6997.686293] LustreError: 180802:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006f5b6922 x1876187581281280/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6997.873830] Lustre: server umount lustre-MDT0001 complete [ 7013.269798] Lustre: server umount lustre-OST0000 complete [ 7023.448356] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7032.348402] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7069.409397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.636195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7069.664263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7069.800311] Lustre: lustre-MDT0000: new disk, initializing [ 7069.870848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.890894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7074.918805] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7087.061366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.240988] Lustre: 185427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7087.321772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7087.325473] Lustre: Skipped 1 previous similar message [ 7087.379948] Lustre: lustre-MDT0001: new disk, initializing [ 7087.473811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7087.554934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7087.591956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7093.203481] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7104.182970] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7111.246964] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7120.834962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.324973] Lustre: lustre-OST0000: new disk, initializing [ 7121.332695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7121.342684] Lustre: 187397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.446899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.111908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7128.139776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7128.300831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7129.640589] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7139.939887] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.753237] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7145.081943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7151.382678] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7151.760254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7159.285526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7159.289318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.300834] LustreError: Skipped 1 previous similar message [ 7159.309074] Lustre: Skipped 1 previous similar message [ 7159.310378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7160.545704] Lustre: server umount lustre-OST0000 complete [ 7169.508354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7169.509109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.512609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7169.512618] Lustre: Skipped 1 previous similar message [ 7169.528588] Lustre: Skipped 2 previous similar messages [ 7171.222759] Lustre: server umount lustre-MDT0000 complete [ 7174.630378] LustreError: 186439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.661871] LustreError: 186439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7177.058705] LustreError: 186224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272199 with bad export cookie 10923538066853572730 [ 7177.075816] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.077225] LustreError: 186224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7177.782747] Lustre: server umount lustre-MDT0001 complete [ 7191.454343] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:03:31 (1789272211) [ 7216.292569] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 7220.323624] Key type lgssc unregistered [ 7220.743617] LNet: 190342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7220.747274] LNetError: 190342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7221.814800] LNet: Removed LNI 192.168.202.149@tcp [ 7223.025494] Key type .llcrypt unregistered [ 7223.027923] Key type ._llcrypt unregistered [ 7242.481615] Key type ._llcrypt registered [ 7242.486055] Key type .llcrypt registered [ 7242.652591] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7244.223360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7244.422513] alg: No test for adler32 (adler32-zlib) [ 7245.700384] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 7246.003782] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 7247.756103] Key type lgssc registered [ 7249.524743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7260.343586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7260.392762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.165396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.460675] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7271.397600] LustreError: 191559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.431472] LustreError: 191559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7276.518120] LustreError: 191560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.596950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.638131] LustreError: 191559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.956205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7288.095512] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7296.444860] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7303.185277] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7311.591431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.084986] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7312.092612] LustreError: 193428:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7312.110238] Lustre: 193428:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7312.395334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.970468] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7326.050170] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7328.813324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.840034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7328.851077] Lustre: Skipped 1 previous similar message [ 7333.350047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7333.365959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.388786] Lustre: Skipped 1 previous similar message [ 7333.400995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.192726] Lustre: server umount lustre-MDT0000 complete [ 7338.978927] LustreError: 192488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7339.030933] LustreError: 192488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7339.823967] LustreError: 191539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272362 with bad export cookie 5883822907032622845 [ 7339.845883] LustreError: 191539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7339.850782] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.140870] Lustre: server umount lustre-MDT0001 complete [ 7354.946659] Lustre: server umount lustre-OST0000 complete [ 7364.908936] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7375.802614] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7414.638537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.900659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7414.958789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7415.107476] Lustre: lustre-MDT0000: new disk, initializing [ 7415.246861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.280805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7421.251754] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7433.017256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.200543] Lustre: 196950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7433.262257] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7433.265375] Lustre: Skipped 1 previous similar message [ 7433.271336] LustreError: 197735:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7433.287893] LustreError: 197735:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7433.301767] Lustre: 197735:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7433.327087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7433.336116] Lustre: Skipped 1 previous similar message [ 7433.438706] Lustre: lustre-MDT0001: new disk, initializing [ 7433.527698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7433.583386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7433.602392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7438.155388] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7449.725935] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7457.745510] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.369317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.735974] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7467.739611] Lustre: Skipped 1 previous similar message [ 7467.745475] LustreError: 198897:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7467.767018] LustreError: 198897:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7467.780410] Lustre: 198897:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7467.792338] Lustre: 198897:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7467.836831] Lustre: lustre-OST0000: new disk, initializing [ 7467.841527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7467.847618] Lustre: 198920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7467.979399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.216041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7473.227774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7473.354995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7474.603721] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7484.784461] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7491.067160] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7491.287830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7495.925129] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7496.131361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7499.238673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.240855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7499.241476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7499.251191] Lustre: Skipped 1 previous similar message [ 7499.274808] Lustre: Skipped 3 previous similar messages [ 7504.358268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.385523] Lustre: Skipped 1 previous similar message [ 7504.757072] Lustre: server umount lustre-OST0000 complete [ 7510.499578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7510.515069] LustreError: Skipped 1 previous similar message [ 7510.542851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.594559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.594493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.713988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7519.725623] Lustre: Skipped 3 previous similar messages [ 7524.320172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7524.495816] Lustre: server umount lustre-MDT0000 complete [ 7524.834990] LustreError: 196963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.845789] LustreError: 196963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7528.770299] LustreError: 197747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272551 with bad export cookie 5883822907032623559 [ 7528.785043] LustreError: 197747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7528.786913] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.312988] Lustre: server umount lustre-MDT0001 complete [ 7539.603941] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:09:20 (1789272560) [ 7557.531411] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7569.212029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.648112] LustreError: 202049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.714302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.175057] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7584.008502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.119509] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7591.865707] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7598.340878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.142066] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7617.912958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7618.326727] Lustre: lustre-OST0001: new disk, initializing [ 7618.334101] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7618.348690] Lustre: 204358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7618.596623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7618.627686] Lustre: Skipped 2 previous similar messages [ 7624.395088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7624.418277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7624.643696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7628.876196] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7644.155199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7653.695570] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7656.408129] Lustre: DEBUG MARKER: ADD OST3 [ 7664.092407] loop: module loaded [ 7664.194672] loop0: detected capacity change from 0 to 204800000 [ 7664.580462] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7664.592742] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7664.616421] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7672.216142] loop0: detected capacity change from 0 to 204800000 [ 7672.956260] Lustre: DEBUG MARKER: STOP OST3 [ 7681.595726] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.081618] Lustre: lustre-OST0003: new disk, initializing [ 7682.098272] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7682.111823] Lustre: 206399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7683.549131] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7683.676112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7683.684532] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7683.724376] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7685.605459] LustreError: 206487:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7686.775661] Lustre: Failing over lustre-OST0003 [ 7687.485765] Lustre: server umount lustre-OST0003 complete [ 7688.674870] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7688.679363] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.687779] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.696098] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7691.913253] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7704.018440] Lustre: DEBUG MARKER: START OST3 again [ 7712.277513] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.696913] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7712.699109] Lustre: Skipped 1 previous similar message [ 7712.710055] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7714.225366] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7714.448745] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7714.449873] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7719.912545] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7744.437926] Lustre: Failing over lustre-OST0003 [ 7744.607065] Lustre: server umount lustre-OST0003 complete [ 7745.507920] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.521072] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.546573] Lustre: Skipped 1 previous similar message [ 7745.554567] LustreError: 203534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.596264] LustreError: 203534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7768.546436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7768.553265] LustreError: Skipped 1 previous similar message [ 7768.561610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.779445] Lustre: server umount lustre-MDT0000 complete [ 7775.698531] LustreError: 202030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272798 with bad export cookie 5883822907032624441 [ 7775.705563] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.714908] LustreError: 202030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7779.308668] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.321099] Lustre: Skipped 6 previous similar messages [ 7790.048988] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7790.561172] Lustre: server umount lustre-MDT0001 complete [ 7795.037518] Lustre: server umount lustre-OST0000 complete [ 7799.209729] Lustre: server umount lustre-OST0001 complete [ 7806.919527] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7816.039753] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7855.123421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.352687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7855.393890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7855.493678] Lustre: lustre-MDT0000: new disk, initializing [ 7855.571426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.592830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7860.230324] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7872.825508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.888536] Lustre: 211819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7872.925464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7872.934948] Lustre: Skipped 1 previous similar message [ 7873.025355] Lustre: lustre-MDT0001: new disk, initializing [ 7873.140409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7873.151233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7878.934179] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7890.225255] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7897.858841] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7906.050614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7906.729580] Lustre: lustre-OST0000: new disk, initializing [ 7906.738833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7906.747854] Lustre: 213789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7908.538750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7908.558054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7908.760035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7916.882292] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7929.560465] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.042476] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7936.460337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.973206] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7942.333745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7949.798394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.809978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7949.818422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7949.818436] Lustre: Skipped 10 previous similar messages [ 7949.822880] Lustre: Skipped 1 previous similar message [ 7949.857878] LustreError: Skipped 1 previous similar message [ 7951.411891] Lustre: server umount lustre-OST0000 complete [ 7963.719895] Lustre: server umount lustre-MDT0000 complete [ 7965.157627] LustreError: 214067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.222573] LustreError: 214067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 7969.113819] LustreError: 211811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272991 with bad export cookie 5883822907032626730 [ 7969.118820] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7969.131095] LustreError: 211811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7969.893725] Lustre: server umount lustre-MDT0001 complete [ 7975.967843] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:16:35 (1789272995) [ 7995.334662] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8010.484302] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8055.468812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.931323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.995889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8056.153948] Lustre: lustre-MDT0000: new disk, initializing [ 8056.263556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8062.755362] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8075.649819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.802105] Lustre: 218427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8076.195958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8082.366279] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8095.187760] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8104.698172] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8112.984520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8113.522793] Lustre: 220399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.668495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.679794] Lustre: Skipped 4 previous similar messages [ 8114.863833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8115.080985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8121.412656] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8131.218642] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8136.773510] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8136.987995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8141.973992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8142.319822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8145.890869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8145.911525] LustreError: Skipped 1 previous similar message [ 8145.917888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8145.941103] Lustre: Skipped 2 previous similar messages [ 8145.945766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8145.954428] Lustre: Skipped 3 previous similar messages [ 8159.205658] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8159.622953] Lustre: server umount lustre-OST0000 complete [ 8171.094593] Lustre: server umount lustre-MDT0000 complete [ 8171.503326] LustreError: 219307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.328849] LustreError: 218421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273200 with bad export cookie 5883822907032627605 [ 8177.342508] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8177.349974] LustreError: 218421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8178.185760] Lustre: server umount lustre-MDT0001 complete [ 8191.505409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.149227] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8206.535054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.645331] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8222.527616] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8229.134686] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8237.990932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.581493] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8253.306697] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8276.971210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8276.977474] LustreError: Skipped 1 previous similar message [ 8276.984368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8276.990304] Lustre: Skipped 7 previous similar messages [ 8279.323462] Lustre: server umount lustre-OST0000 complete [ 8292.064569] Lustre: server umount lustre-MDT0000 complete [ 8295.753954] LustreError: 224464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273318 with bad export cookie 5883822907032628494 [ 8295.760124] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8295.769832] LustreError: 224464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8296.478838] Lustre: server umount lustre-MDT0001 complete [ 8396.101553] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8399.400640] Key type lgssc unregistered [ 8399.879739] LNet: 226303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8399.885427] LNetError: 226303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8399.922369] LNet: Removed LNI 192.168.202.149@tcp [ 8400.955164] Key type .llcrypt unregistered [ 8400.957111] Key type ._llcrypt unregistered [ 8434.824914] Key type ._llcrypt registered [ 8434.829590] Key type .llcrypt registered [ 8435.022292] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8451.196372] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8452.766977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8452.955471] alg: No test for adler32 (adler32-zlib) [ 8454.145949] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8454.493154] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 8456.232604] Key type lgssc registered [ 8457.643411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8494.901138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8494.988698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.325434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8496.349242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8496.418741] Lustre: lustre-MDT0000: new disk, initializing [ 8496.533183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.587068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8502.272800] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8516.304923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8516.498592] Lustre: 229661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8516.619732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8516.629997] Lustre: Skipped 1 previous similar message [ 8516.782900] Lustre: lustre-MDT0001: new disk, initializing [ 8517.036711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8517.140683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8517.179188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8523.468557] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8538.540387] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.318475] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8557.470678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8557.703561] Lustre: lustre-OST0000: new disk, initializing [ 8557.710989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8557.718312] Lustre: 231631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8557.832835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8564.491251] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8566.366521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8566.385815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8566.558179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8575.438993] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.893079] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8582.244190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8587.351252] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8587.672535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8592.357678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8592.363596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.380483] LustreError: Skipped 1 previous similar message [ 8592.382813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8592.409454] Lustre: Skipped 1 previous similar message [ 8592.447644] Lustre: Skipped 1 previous similar message [ 8595.897859] Lustre: server umount lustre-OST0000 complete [ 8602.600093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8602.616484] Lustre: Skipped 1 previous similar message [ 8602.618216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8602.634916] Lustre: Skipped 1 previous similar message [ 8606.690685] Lustre: server umount lustre-MDT0000 complete [ 8610.456764] LustreError: 229655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273633 with bad export cookie 7451658700244068082 [ 8610.460273] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8610.475472] LustreError: 229655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8610.944418] Lustre: server umount lustre-MDT0001 complete [ 8629.681627] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8641.610476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.322242] LustreError: 234567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.351972] LustreError: 234567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8642.422433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.649637] LustreError: 234568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.313101] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8652.773279] LustreError: 234568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.888956] LustreError: 234567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8660.075692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.789208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8665.726809] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8668.491954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8675.667514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8676.165882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.468537] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8696.436597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.615676] Lustre: lustre-OST0001: new disk, initializing [ 8696.619672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8696.625238] Lustre: 236874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8696.739043] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8706.183926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8706.191843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8706.283054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8706.369493] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8720.321358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8733.332972] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8748.311661] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:29:28 (1789273768) [ 8752.630029] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.640032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8752.668386] Lustre: Skipped 2 previous similar messages [ 8752.678225] Lustre: Skipped 1 previous similar message [ 8757.731496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.735133] Lustre: Skipped 2 previous similar messages [ 8758.266021] Lustre: server umount lustre-MDT0000 complete [ 8762.859958] LustreError: 234564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.884844] LustreError: 234564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8763.733962] LustreError: 234547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273786 with bad export cookie 7451658700244068964 [ 8763.738541] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8763.742823] LustreError: 234547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8764.834866] Lustre: server umount lustre-MDT0001 complete [ 8781.229211] Lustre: server umount lustre-OST0000 complete [ 8783.328115] Lustre: 227883:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273790/real 1789273790] req@ffff99bc817a6680 x1876189225542400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8783.414217] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.460294] Lustre: Skipped 1 previous similar message [ 8784.355882] Lustre: 227885:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273791/real 1789273791] req@ffff99bc817a5880 x1876189225542656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8786.641118] Lustre: server umount lustre-OST0001 complete [ 8795.815532] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8809.529951] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8858.762274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8859.087740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8859.141138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8859.257879] Lustre: lustre-MDT0000: new disk, initializing [ 8859.338579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8859.359940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8864.541772] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8876.160840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8876.299260] Lustre: 240827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8876.406694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8876.413672] Lustre: Skipped 1 previous similar message [ 8876.537700] Lustre: lustre-MDT0001: new disk, initializing [ 8876.761702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8876.774541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8881.794039] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8892.695117] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8899.419173] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8907.300156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8907.808877] Lustre: lustre-OST0000: new disk, initializing [ 8907.815129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8907.826935] Lustre: 242796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8909.884699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8909.895914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8910.153883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8916.492932] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8926.894717] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8931.259356] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8931.535382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8936.966621] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8937.202697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8941.030900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8941.037208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.039945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8941.039954] Lustre: Skipped 1 previous similar message [ 8941.042832] LustreError: Skipped 1 previous similar message [ 8941.088232] Lustre: Skipped 1 previous similar message [ 8945.670330] Lustre: server umount lustre-OST0000 complete [ 8951.267355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.288439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8951.302558] Lustre: Skipped 1 previous similar message [ 8951.332157] Lustre: Skipped 2 previous similar messages [ 8957.030547] Lustre: server umount lustre-MDT0000 complete [ 8960.486959] LustreError: 242805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.516318] LustreError: 242805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8961.489187] LustreError: 240819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273984 with bad export cookie 7451658700244071106 [ 8961.498196] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.503455] LustreError: 240819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8961.739417] Lustre: server umount lustre-MDT0001 complete [ 8971.534392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.269157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.277336] Lustre: Skipped 2 previous similar messages [ 8976.995263] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8977.383536] LustreError: 245015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.395148] LustreError: 245015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8985.579722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.747516] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8999.277596] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.521347] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9012.859086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9023.420413] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9035.698229] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9180.653697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9180.664305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.704444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9180.711122] Lustre: Skipped 2 previous similar messages [ 9184.759967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9186.849674] Lustre: server umount lustre-OST0000 complete [ 9190.883130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9197.070200] Lustre: server umount lustre-MDT0000 complete [ 9200.101986] LustreError: 245015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.126397] LustreError: 245015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9201.621444] LustreError: 244996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274224 with bad export cookie 7451658700244071988 [ 9201.636646] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.215272] Lustre: server umount lustre-MDT0001 complete [ 9306.065137] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 9310.226271] Key type lgssc unregistered [ 9310.482835] LNet: 248620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9310.494758] LNetError: 248620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9310.534764] LNet: Removed LNI 192.168.202.149@tcp [ 9311.420144] Key type .llcrypt unregistered [ 9311.425417] Key type ._llcrypt unregistered [ 9338.063123] Key type ._llcrypt registered [ 9338.065448] Key type .llcrypt registered [ 9338.159986] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9360.342166] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9362.080329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9362.104034] alg: No test for adler32 (adler32-zlib) [ 9363.312463] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 9363.778426] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 9365.576639] Key type lgssc registered [ 9367.550825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9422.601467] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9435.841868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9435.874059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.210956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9437.250640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9437.341237] Lustre: lustre-MDT0000: new disk, initializing [ 9437.415842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9437.474903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9441.788572] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9455.191942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.400154] Lustre: 252701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9455.465152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9455.469549] Lustre: Skipped 1 previous similar message [ 9455.551400] Lustre: lustre-MDT0001: new disk, initializing [ 9455.680734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9455.727159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9455.746620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9460.204701] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9465.434798] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9471.769576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9472.043853] Lustre: lustre-OST0000: new disk, initializing [ 9472.053362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9472.064835] Lustre: 254327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9472.181490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.184415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9474.195644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9474.292618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9477.874537] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9488.143215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.355636] Lustre: lustre-OST0001: new disk, initializing [ 9488.359102] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9488.363537] Lustre: 255201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9488.452405] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9494.642427] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9497.188718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9497.211305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9497.278539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9506.975100] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9514.605517] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9527.266392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9527.277846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.297472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9528.293562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9528.307813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.338257] Lustre: Skipped 1 previous similar message [ 9533.623306] Lustre: server umount lustre-MDT0000 complete [ 9538.010687] LustreError: 254326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274560 with bad export cookie 6701034861842349972 [ 9538.019721] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.028982] LustreError: 254326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9538.598914] Lustre: server umount lustre-MDT0001 complete [ 9554.195655] Lustre: server umount lustre-OST0000 complete [ 9554.912315] Lustre: 250203:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789274561/real 1789274561] req@ffff99bc898cc700 x1876190178652032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789274577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9554.953317] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.972596] Lustre: Skipped 1 previous similar message [ 9559.327636] Lustre: server umount lustre-OST0001 complete [ 9573.714621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9582.633444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9589.985094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9599.820301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9621.747190] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9632.699387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9632.715887] Lustre: lustre-MDT0000: reset Object Index mappings [ 9633.255625] LustreError: 260213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.273156] LustreError: 260213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9633.337157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9637.700643] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9638.372949] LustreError: 260214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9643.491055] LustreError: 260213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.093374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.143927] Lustre: lustre-MDT0001: reset Object Index mappings [ 9647.849606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9652.928642] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9656.108223] Lustre: 261354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9663.026185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.071148] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9672.772520] LustreError: 261710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.796192] LustreError: 261712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.849195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9679.212717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9679.597061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.604551] Lustre: Skipped 1 previous similar message [ 9685.030058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9685.831595] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9693.251413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9697.486677] Lustre: 263226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9703.921725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9703.936823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.961426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9703.976178] Lustre: Skipped 3 previous similar messages [ 9704.715251] Lustre: server umount lustre-MDT0000 complete [ 9705.449525] LustreError: 260946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.473357] LustreError: 260946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9708.124281] LustreError: 261356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274730 with bad export cookie 6701034861842356762 [ 9708.132054] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9708.139265] LustreError: 261356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9708.582916] Lustre: server umount lustre-MDT0001 complete [ 9722.688901] Lustre: server umount lustre-OST0000 complete [ 9726.944423] Lustre: 250203:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789274733/real 1789274733] req@ffff99bc8303fb80 x1876190178700928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789274749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9726.965669] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.974616] Lustre: Skipped 3 previous similar messages [ 9727.203600] Lustre: server umount lustre-OST0001 complete [ 9751.144935] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:46:11 (1789274771) [ 9768.157983] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9780.610738] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9825.733051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.074595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9826.122234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9826.264610] Lustre: lustre-MDT0000: new disk, initializing [ 9826.395427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.429577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9830.551083] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9841.433912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9841.623878] Lustre: 267482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9841.718613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9841.722742] Lustre: Skipped 1 previous similar message [ 9841.854636] Lustre: lustre-MDT0001: new disk, initializing [ 9842.071301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9842.101769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9847.854112] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9859.288828] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9866.631486] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9874.955575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.631614] Lustre: lustre-OST0000: new disk, initializing [ 9875.642372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9875.663262] Lustre: 269450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9877.467775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9877.505698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9877.745064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9883.995635] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9894.763689] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9899.712917] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9899.940298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9904.964342] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9905.241967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9908.194226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9908.200464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.217588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9913.527939] Lustre: server umount lustre-OST0000 complete [ 9918.956254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9918.956344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.956351] Lustre: Skipped 1 previous similar message [ 9924.066613] LustreError: 268320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.083698] LustreError: 268320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9925.968898] Lustre: server umount lustre-MDT0000 complete [ 9929.792821] LustreError: 267472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274952 with bad export cookie 6701034861842358855 [ 9929.802278] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.810487] LustreError: 267472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9930.237451] Lustre: server umount lustre-MDT0001 complete [ 9950.497592] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9960.839916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.292827] LustreError: 272386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.311299] LustreError: 272386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9961.363642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.370795] Lustre: Skipped 2 previous similar messages [ 9965.493769] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9976.767040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.669561] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9987.671676] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9997.745643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.896980] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10016.073479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10016.286262] Lustre: lustre-OST0001: new disk, initializing [10016.292110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10016.298274] Lustre: 274694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10017.731716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10017.753574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10018.024589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10023.864282] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10034.381376] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10038.311985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10044.853194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10044.950316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10053.602528] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.620069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10053.624672] Lustre: Skipped 5 previous similar messages [10053.631297] Lustre: Skipped 4 previous similar messages [10059.950477] Lustre: server umount lustre-MDT0000 complete [10063.857172] LustreError: 272387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.873021] LustreError: 272387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10064.602671] LustreError: 272367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275087 with bad export cookie 6701034861842359737 [10064.604584] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10064.613973] LustreError: 272367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10064.935095] Lustre: server umount lustre-MDT0001 complete [10079.884251] Lustre: server umount lustre-OST0000 complete [10094.307278] Lustre: server umount lustre-OST0001 complete [10112.726614] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10124.731234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.566566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10125.571381] Lustre: Skipped 3 previous similar messages [10130.006685] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10138.120670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.435385] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10146.641625] Lustre: 278892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10153.124539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10160.238606] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10169.500931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.848142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10170.900837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10177.123341] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10184.578771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10188.379435] Lustre: 280771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10193.714939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10194.001237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10203.650157] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:53:43 (1789275223) [10220.521723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10220.530451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.552515] Lustre: Skipped 1 previous similar message [10220.570062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10220.573536] Lustre: Skipped 4 previous similar messages [10223.944340] Lustre: server umount lustre-MDT0000 complete [10227.180264] LustreError: 277753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10227.193385] LustreError: 277753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10227.844810] LustreError: 277735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275250 with bad export cookie 6701034861842366268 [10227.857497] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10227.867202] LustreError: 277735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10234.394459] Lustre: server umount lustre-MDT0001 complete [10253.280767] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10253.408709] Lustre: server umount lustre-OST0000 complete [10263.817808] Lustre: server umount lustre-OST0001 complete [10271.046151] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10282.794842] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10339.029461] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10350.748053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.029444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10351.055318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10351.112492] Lustre: lustre-MDT0000: new disk, initializing [10351.219643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10356.320283] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10366.832802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.095286] Lustre: 285017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10367.806917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10372.954348] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10390.358840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.651547] Lustre: 286649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10390.747164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10390.752533] Lustre: Skipped 5 previous similar messages [10392.810147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10399.072985] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10412.521931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10412.699284] Lustre: 287522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10414.293578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10414.306713] Lustre: Skipped 1 previous similar message [10414.465469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10420.155506] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10429.469567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10433.512310] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10433.517339] Lustre: Skipped 1 previous similar message [10449.138247] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:57:49 (1789275469) [10453.731214] Lustre: server umount lustre-MDT0000 complete [10454.505376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10454.513805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.523747] Lustre: Skipped 5 previous similar messages [10457.159551] LustreError: 285010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275479 with bad export cookie 6701034861842373905 [10457.160746] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10457.169807] LustreError: 285010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10457.737418] Lustre: server umount lustre-MDT0001 complete [10472.372286] Lustre: server umount lustre-OST0000 complete [10476.500904] Lustre: 250205:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789275483/real 1789275483] req@ffff99bc87b22680 x1876190178967168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789275499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10477.522942] Lustre: server umount lustre-OST0001 complete [10486.459683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.823646] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.839896] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10491.343417] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10499.718621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.205883] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10512.641478] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.436505] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.313702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10534.340543] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10543.749160] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10563.704047] loop1: detected capacity change from 0 to 204800000 [10563.912734] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10563.925837] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10563.937188] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10569.341526] loop1: detected capacity change from 0 to 204800000 [10574.568562] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.639991] LustreError: 289754:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10713: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10574.649826] LustreError: 289754:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10713 [10574.653669] LustreError: 289754:0:(mgs_handler.c:517:mgs_target_reg()) fs10713-MDT0001: Error checking target index: rc = -13 [10574.662509] LustreError: fs10713-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10574.672465] LustreError: 293485:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10574.679032] LustreError: 293485:0:(tgt_mount.c:2132:server_put_super()) no obd fs10713-MDT0001 [10574.684774] LustreError: 293485:0:(tgt_mount.c:116:server_deregister_mount()) fs10713-MDT0001 not registered [10574.723410] Lustre: server umount fs10713-MDT0001 complete [10574.727148] LustreError: 293485:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10583.658722] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.787701] Lustre: Setting parameter fs10713-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10713-MDT0001 [10583.840906] Lustre: srv-fs10713-MDT0001: No data found on store. Initialize space. [10583.845260] Lustre: Skipped 4 previous similar messages [10583.940768] Lustre: fs10713-MDT0001: new disk, initializing [10583.945896] Lustre: Skipped 3 previous similar messages [10584.105617] Lustre: srv-fs10713-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10584.121639] Lustre: 293952:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10713-MDT0001-mdtlov: can't create new llog: rc = -115 [10584.133278] LustreError: 293952:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10713-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10588.858878] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10595.737873] Lustre: server umount fs10713-MDT0001 complete [10609.310573] loop1: detected capacity change from 0 to 204800000 [10611.698733] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10615.842948] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.396955] Lustre: 295519:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10713-MDT0001-mdtlov: can't create new llog: rc = -115 [10616.404765] LustreError: 295519:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10713-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10621.450888] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10628.393080] Lustre: server umount fs10713-MDT0001 complete [10647.255318] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [10647.264233] Lustre: Skipped 6 previous similar messages [10647.548278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.590398] Lustre: Skipped 6 previous similar messages [10652.429951] Lustre: server umount lustre-MDT0000 complete [10660.903446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.005557] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10666.486468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10667.197654] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10675.561700] loop1: detected capacity change from 0 to 204800000 [10682.360826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.872323] Lustre: 297983:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10713-MDT0001-mdtlov: can't create new llog: rc = -115 [10682.879843] LustreError: 297983:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10713-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10687.853154] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10693.045796] Lustre: server umount fs10713-MDT0001 complete [10722.794879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10722.808167] LustreError: Skipped 1 previous similar message [10722.812194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10722.854431] Lustre: Skipped 4 previous similar messages [10727.884994] Lustre: server umount lustre-OST0000 complete [10739.127558] Lustre: server umount lustre-MDT0000 complete [10743.668193] LustreError: 289747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275766 with bad export cookie 6701034861842376586 [10743.671326] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10743.696133] LustreError: 289747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10744.109243] Lustre: server umount lustre-MDT0001 complete [10760.146339] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [10764.627874] Key type lgssc unregistered [10765.095632] LNet: 300329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10765.109803] LNetError: 300329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10765.147787] LNet: Removed LNI 192.168.202.149@tcp [10766.801146] Key type .llcrypt unregistered [10766.803689] Key type ._llcrypt unregistered [10794.769620] Key type ._llcrypt registered [10794.774590] Key type .llcrypt registered [10794.920959] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10796.176091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10796.273643] alg: No test for adler32 (adler32-zlib) [10797.936038] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [10798.255032] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [10799.952818] Key type lgssc registered [10801.275839] Lustre: Echo OBD driver; http://www.lustre.org/ [10809.533544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10809.545855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10811.083321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10816.469496] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10820.594716] LustreError: 301598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10820.634893] LustreError: 301598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10825.704915] LustreError: 301599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.294512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10826.872982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10832.347809] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10840.609458] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10846.605972] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10853.450666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.000296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.051939] LustreError: 303429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.546434] LustreError: 303430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.722223] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10866.672140] LustreError: 303560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.069481] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10875.167288] LustreError: 303427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.190276] LustreError: 303427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10892.281504] LustreError: 303427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.306966] LustreError: 303427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10895.513318] loop1: detected capacity change from 0 to 204800000 [10895.782292] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10895.796271] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10895.813860] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10900.783368] loop1: detected capacity change from 0 to 204800000 [10908.156605] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10908.492801] Lustre: fs9559-OST0000: new disk, initializing [10908.507780] Lustre: srv-fs9559-OST0000: No data found on store. Initialize space. [10908.534040] Lustre: 305338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10908.660686] Lustre: fs9559-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10916.924853] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10926.468827] Lustre: server umount fs9559-OST0000 complete [10926.725668] LustreError: 303429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10926.739512] LustreError: 303429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10938.360701] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10939.191996] loop1: detected capacity change from 0 to 204800000 [10945.509572] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10946.014899] Lustre: fs9559-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10953.207640] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10953.717422] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10962.557094] loop2: detected capacity change from 0 to 204800000 [10962.780859] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10962.811474] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10962.837742] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10968.968315] loop2: detected capacity change from 0 to 204800000 [10975.188590] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10975.397280] LustreError: fs9559-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10975.419587] LustreError: fs9559-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10975.442353] LustreError: 308376:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10975.454721] LustreError: 308376:0:(tgt_mount.c:2132:server_put_super()) no obd fs9559-OST0001 [10975.463135] LustreError: 308376:0:(tgt_mount.c:116:server_deregister_mount()) fs9559-OST0001 not registered [10975.517698] Lustre: server umount fs9559-OST0001 complete [10975.521810] LustreError: 308376:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10987.487964] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10987.579805] Lustre: fs9559-OST0001: new disk, initializing [10987.582976] Lustre: srv-fs9559-OST0001: No data found on store. Initialize space. [10987.588663] Lustre: 308928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10987.644209] Lustre: fs9559-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10993.277218] LustreError: 303430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.304708] LustreError: 303430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10994.459167] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11003.913814] Lustre: server umount fs9559-OST0000 complete [11013.836135] Lustre: server umount fs9559-OST0001 complete [11027.805213] loop1: detected capacity change from 0 to 204800000 [11034.931894] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11035.228810] Lustre: fs9559-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11042.717122] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11051.156364] loop2: detected capacity change from 0 to 204800000 [11058.718242] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11068.211356] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11082.711258] Lustre: server umount fs9559-OST0000 complete [11091.956880] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11094.074210] Lustre: server umount fs9559-OST0001 complete [11107.307212] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11112.420863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11112.421532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.431575] LustreError: Skipped 1 previous similar message [11112.435612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11112.450831] Lustre: Skipped 1 previous similar message [11112.461664] Lustre: Skipped 1 previous similar message [11116.338330] Lustre: server umount lustre-OST0000 complete [11122.678316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11122.700802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.708878] Lustre: Skipped 1 previous similar message [11127.236604] Lustre: server umount lustre-MDT0000 complete [11127.777385] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.798318] LustreError: 304103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 71 previous similar messages [11132.252347] LustreError: 301578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276155 with bad export cookie 11973772053036268537 [11132.265116] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11132.266736] LustreError: 301578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11132.859949] Lustre: server umount lustre-MDT0001 complete [11147.119927] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11150.593630] Key type lgssc unregistered [11151.040091] LNet: 314853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11151.059515] LNetError: 314853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11151.093600] LNet: Removed LNI 192.168.202.149@tcp [11152.331635] Key type .llcrypt unregistered [11152.336031] Key type ._llcrypt unregistered [11189.754372] Key type ._llcrypt registered [11189.759325] Key type .llcrypt registered [11189.894195] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11209.202226] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11210.630068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11210.707062] alg: No test for adler32 (adler32-zlib) [11211.827298] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11212.200649] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11213.904974] Key type lgssc registered [11215.294036] Lustre: Echo OBD driver; http://www.lustre.org/ [11257.288703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11257.335497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11258.675317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11258.715849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11258.818411] Lustre: lustre-MDT0000: new disk, initializing [11258.922872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11258.939593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11264.541127] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11277.315660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.497472] Lustre: 318711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11277.590740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11277.597283] Lustre: Skipped 1 previous similar message [11277.717665] Lustre: lustre-MDT0001: new disk, initializing [11277.884132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11277.972065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11277.988954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11283.761725] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11295.700471] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11302.337581] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11309.797295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11310.169702] Lustre: lustre-OST0000: new disk, initializing [11310.173181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11310.179136] Lustre: 320679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11310.284838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11316.315848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11316.328473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11316.432900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11317.362618] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11327.705200] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11330.642890] Lustre: Mounted lustre-client - version 2.17.57_105_g2595dd0 [11331.643731] Lustre: Unmounted lustre-client [11342.308233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11342.314906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.322362] LustreError: Skipped 1 previous similar message [11342.323886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11342.348689] Lustre: Skipped 1 previous similar message [11345.893658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11345.913295] Lustre: Skipped 2 previous similar messages [11348.383658] Lustre: server umount lustre-OST0000 complete [11354.599776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11354.606520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11354.631525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11356.131437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.281371] Lustre: server umount lustre-MDT0000 complete [11363.747878] LustreError: 318703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276386 with bad export cookie 485150848050135003 [11363.766282] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11364.125314] Lustre: server umount lustre-MDT0001 complete [11375.980486] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11379.543496] Key type lgssc unregistered [11381.153871] LNet: 1 local NIs in recovery (showing 1): 192.168.202.149@tcp [11381.926294] LNet: 322843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11381.966250] LNetError: 322843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11461.868625] LNet: Removed LNI 192.168.202.149@tcp [11462.878059] Key type .llcrypt unregistered [11462.880093] Key type ._llcrypt unregistered [11482.180317] Key type ._llcrypt registered [11482.183887] Key type .llcrypt registered [11482.269709] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11497.865913] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11499.106278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11499.215050] alg: No test for adler32 (adler32-zlib) [11500.271232] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11500.454429] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11502.160882] Key type lgssc registered [11503.476603] Lustre: Echo OBD driver; http://www.lustre.org/ [11537.683387] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:15:57 (1789276557) [11545.319139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11545.418766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.736471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11546.767736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11546.890102] Lustre: lustre-MDT0000: new disk, initializing [11546.994433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11547.023261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11553.036629] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11565.221680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11565.376816] Lustre: 326070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11565.494629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11565.499959] Lustre: Skipped 1 previous similar message [11565.648705] Lustre: lustre-MDT0001: new disk, initializing [11565.803817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11565.867674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11565.880632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11571.493210] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11582.167971] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.736879] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11595.373306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.726864] Lustre: lustre-OST0000: new disk, initializing [11595.732884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11595.742752] Lustre: 328034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11595.834326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11602.800249] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11604.589258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11604.608886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11604.735168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11612.861643] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11630.563988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11630.565793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.578572] LustreError: Skipped 1 previous similar message [11630.584068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11630.598473] Lustre: Skipped 1 previous similar message [11634.597862] Lustre: server umount lustre-OST0000 complete [11640.808503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.813978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.820538] Lustre: Skipped 1 previous similar message [11640.832287] Lustre: Skipped 2 previous similar messages [11645.479960] Lustre: server umount lustre-MDT0000 complete [11649.657583] LustreError: 326061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276672 with bad export cookie 17049637756850231439 [11649.667812] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.674747] LustreError: 326061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11650.084372] Lustre: server umount lustre-MDT0001 complete [11663.125939] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11666.453506] Key type lgssc unregistered [11666.792281] LNet: 329943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11666.811204] LNetError: 329943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11666.828809] LNet: Removed LNI 192.168.202.149@tcp [11667.780504] Key type .llcrypt unregistered [11667.783192] Key type ._llcrypt unregistered [11689.335745] Key type ._llcrypt registered [11689.338528] Key type .llcrypt registered [11689.463722] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11690.666182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11690.774201] alg: No test for adler32 (adler32-zlib) [11691.833583] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11692.035196] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11693.696168] Key type lgssc registered [11694.819625] Lustre: Echo OBD driver; http://www.lustre.org/ [11702.662611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11702.674319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.180897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11708.878319] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11713.525401] LustreError: 331216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11713.548532] LustreError: 331216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11718.625863] LustreError: 331215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.694728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.119213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11723.873550] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11732.996791] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11739.792533] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11748.089977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11748.512284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11755.280129] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11763.614646] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.811442] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11777.189423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11784.698518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.722201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11784.732642] Lustre: Skipped 2 previous similar messages [11784.762841] Lustre: Skipped 2 previous similar messages [11785.905562] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [11789.800484] LustreError: 331215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11789.814166] Lustre: server umount lustre-MDT0000 complete [11789.837547] LustreError: 331215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11794.926579] LustreError: 333872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.956894] LustreError: 333872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11795.362957] LustreError: 331197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276818 with bad export cookie 9665984783213153798 [11795.375356] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.375619] LustreError: 331197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11796.142586] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.49@tcp (stopping) [11800.037289] LustreError: 333054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.038406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.069751] LustreError: 333054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11800.107863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11801.837824] Lustre: server umount lustre-MDT0001 complete [11806.358830] Lustre: server umount lustre-OST0000 complete [11817.697447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.275906] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11825.017269] Lustre: 335396:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.149@tcp [11827.105778] Lustre: server umount MGS complete [11836.548870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11837.354602] LustreError: 335968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.382205] LustreError: 335968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11837.447813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.942669] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11852.674527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.384730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11858.572316] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11867.827503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11868.442918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11875.071837] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11883.641319] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.933993] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11893.811386] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.950846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11903.457989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11903.460692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.474789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11903.474798] Lustre: Skipped 1 previous similar message [11903.501501] Lustre: Skipped 1 previous similar message [11909.355041] Lustre: server umount lustre-OST0000 complete [11913.699629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.712865] Lustre: Skipped 1 previous similar message [11913.713220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11913.729041] Lustre: Skipped 4 previous similar messages [11919.332164] Lustre: server umount lustre-MDT0000 complete [11923.941122] LustreError: 335968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.964846] LustreError: 335968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11924.308477] LustreError: 335949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276947 with bad export cookie 9665984783213154645 [11924.319352] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11924.325709] LustreError: 335949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11925.142749] Lustre: server umount lustre-MDT0001 complete [11937.680613] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11940.608028] Key type lgssc unregistered [11940.973787] LNet: 339404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11940.985661] LNetError: 339404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11941.003143] LNet: Removed LNI 192.168.202.149@tcp [11941.993480] Key type .llcrypt unregistered [11941.999207] Key type ._llcrypt unregistered [11963.392419] Key type ._llcrypt registered [11963.394971] Key type .llcrypt registered [11963.563814] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11981.584539] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11983.064335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11983.157133] alg: No test for adler32 (adler32-zlib) [11984.195542] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11984.469814] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11986.170289] Key type lgssc registered [11987.558943] Lustre: Echo OBD driver; http://www.lustre.org/ [12030.321290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12030.345214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.617731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12031.647309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12031.715762] Lustre: lustre-MDT0000: new disk, initializing [12031.832664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12031.860028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12036.030716] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12047.522457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.632571] Lustre: 342563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12047.691694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12047.695886] Lustre: Skipped 1 previous similar message [12047.767430] Lustre: lustre-MDT0001: new disk, initializing [12047.852238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12047.892865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12047.908927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12052.499344] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12062.574917] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12069.008344] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12076.350882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12076.741878] Lustre: lustre-OST0000: new disk, initializing [12076.747732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12076.758372] Lustre: 344531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12076.882588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.804362] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12085.839946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12085.848608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12086.016891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12093.240621] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12098.787311] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12099.039854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12105.020095] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12105.216785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12111.842852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.848369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12111.864940] Lustre: Skipped 1 previous similar message [12111.871495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12111.883758] Lustre: Skipped 1 previous similar message [12113.835651] Lustre: server umount lustre-OST0000 complete [12119.521949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12119.534694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.549878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12122.084260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.084486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12122.123164] Lustre: Skipped 1 previous similar message [12126.743377] Lustre: server umount lustre-MDT0000 complete [12130.786428] LustreError: 342556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277153 with bad export cookie 12071550609331191092 [12130.801401] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12131.191983] Lustre: server umount lustre-MDT0001 complete [12140.448497] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:26:00 (1789277160) [12157.220897] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [12168.316707] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12205.808362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.058746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12206.085737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12206.152805] Lustre: lustre-MDT0000: new disk, initializing [12206.241184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12206.267923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12210.973857] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12222.639301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.772727] Lustre: 349294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12222.850026] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12222.863467] Lustre: Skipped 1 previous similar message [12222.982066] Lustre: lustre-MDT0001: new disk, initializing [12223.119050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.170463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12223.191712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12228.148819] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12237.675439] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12244.398707] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12252.636516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.028545] Lustre: lustre-OST0000: new disk, initializing [12253.032752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12253.045809] Lustre: 351261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12253.147356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.639905] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12262.990378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12262.998677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12263.128390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12271.393097] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12309.475310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12309.482072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12309.488566] LustreError: Skipped 1 previous similar message [12309.498983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12309.510567] Lustre: Skipped 1 previous similar message [12309.530932] Lustre: Skipped 1 previous similar message [12314.594797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12314.604042] Lustre: Skipped 1 previous similar message [12323.297672] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12323.567138] Lustre: server umount lustre-OST0000 complete [12329.959940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.966751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12329.984339] Lustre: Skipped 1 previous similar message [12329.985287] Lustre: Skipped 2 previous similar messages [12333.608950] Lustre: server umount lustre-MDT0000 complete [12335.077535] LustreError: 349299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.109899] LustreError: 349299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12338.456896] LustreError: 349284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277361 with bad export cookie 12071550609331191974 [12338.478923] LustreError: 349284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12338.481339] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12339.056815] Lustre: server umount lustre-MDT0001 complete [12350.874490] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12353.649721] Key type lgssc unregistered [12353.984535] LNet: 353437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12353.989271] LNetError: 353437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12354.015290] LNet: Removed LNI 192.168.202.149@tcp [12354.912549] Key type .llcrypt unregistered [12354.914589] Key type ._llcrypt unregistered [12454.860394] Key type ._llcrypt registered [12454.863218] Key type .llcrypt registered [12454.969110] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12455.381354] Key type .llcrypt unregistered [12455.387150] Key type ._llcrypt unregistered [12538.408886] Key type ._llcrypt registered [12538.411413] Key type .llcrypt registered [12538.541812] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12538.953352] Key type .llcrypt unregistered [12538.956773] Key type ._llcrypt unregistered [12571.970408] Key type ._llcrypt registered [12571.976866] Key type .llcrypt registered [12572.128829] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12572.624401] Key type .llcrypt unregistered [12572.626181] Key type ._llcrypt unregistered [12603.242142] Key type ._llcrypt registered [12603.244310] Key type .llcrypt registered [12603.333105] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12603.580641] Key type .llcrypt unregistered [12603.582946] Key type ._llcrypt unregistered [12687.478306] Key type ._llcrypt registered [12687.485752] Key type .llcrypt registered [12687.714481] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12688.058640] Key type .llcrypt unregistered [12688.066314] Key type ._llcrypt unregistered [12721.452742] Key type ._llcrypt registered [12721.454993] Key type .llcrypt registered [12721.564273] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12721.857465] Key type .llcrypt unregistered [12721.860729] Key type ._llcrypt unregistered [12752.661995] Key type ._llcrypt registered [12752.665376] Key type .llcrypt registered [12752.762146] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12753.101425] Key type .llcrypt unregistered [12753.109984] Key type ._llcrypt unregistered [12784.530904] Key type ._llcrypt registered [12784.533907] Key type .llcrypt registered [12784.681326] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12784.973654] Key type .llcrypt unregistered [12784.976541] Key type ._llcrypt unregistered [12815.767658] Key type ._llcrypt registered [12815.769738] Key type .llcrypt registered [12815.860944] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12816.219548] Key type .llcrypt unregistered [12816.224865] Key type ._llcrypt unregistered [12851.761802] Key type ._llcrypt registered [12851.764740] Key type .llcrypt registered [12851.944842] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12852.647415] Key type .llcrypt unregistered [12852.649146] Key type ._llcrypt unregistered [12926.066745] Key type ._llcrypt registered [12926.069341] Key type .llcrypt registered [12926.168892] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12926.565511] Key type .llcrypt unregistered [12926.568529] Key type ._llcrypt unregistered [12963.570754] Key type ._llcrypt registered [12963.572451] Key type .llcrypt registered [12963.811347] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12964.163539] Key type .llcrypt unregistered [12964.165597] Key type ._llcrypt unregistered [12998.480607] Key type ._llcrypt registered [12998.482831] Key type .llcrypt registered [12998.602347] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12998.983686] Key type .llcrypt unregistered [12998.985323] Key type ._llcrypt unregistered [13056.053869] Key type ._llcrypt registered [13056.057162] Key type .llcrypt registered [13056.159968] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13070.259517] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13071.198684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13071.248669] alg: No test for adler32 (adler32-zlib) [13072.511869] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [13072.709626] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [13074.360211] Key type lgssc registered [13075.314139] Lustre: Echo OBD driver; http://www.lustre.org/ [13112.566713] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13112.594866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.108981] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13114.196788] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13114.332739] Lustre: tst250fs-MDT0000: new disk, initializing [13114.497377] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13114.525464] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13120.873774] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13135.238975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13135.488570] Lustre: 373735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13135.613284] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13135.626331] Lustre: Skipped 1 previous similar message [13135.774447] Lustre: tst250fs-MDT0001: new disk, initializing [13135.960965] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13136.039507] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13136.072127] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13143.025689] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13148.594885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13156.290891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13156.656708] Lustre: tst250fs-OST0000: new disk, initializing [13156.661662] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13156.668577] Lustre: 375367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13156.800526] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13159.006053] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13159.017068] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13159.190116] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13163.540951] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13176.243995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13176.417536] Lustre: tst250fs-OST0001: new disk, initializing [13176.424996] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13176.437620] Lustre: 376240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13176.587216] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13184.042614] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13185.729616] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13185.741262] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13185.840154] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13190.061840] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13192.094551] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13192.103924] Lustre: Skipped 1 previous similar message [13212.651234] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13212.657082] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13212.671357] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13216.737777] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13216.739722] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13216.769816] Lustre: Skipped 1 previous similar message [13218.624615] Lustre: server umount tst250fs-MDT0000 complete [13222.619104] LustreError: 373728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278245 with bad export cookie 9452066251963061862 [13222.639548] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13222.639822] LustreError: 373728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13222.932238] Lustre: server umount tst250fs-MDT0001 complete [13237.424452] Lustre: server umount tst250fs-OST0000 complete [13252.083777] Lustre: server umount tst250fs-OST0001 complete [13271.574796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13281.420431] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13315.356706] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:45:36 (1789278336) [13317.146253] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13336.773624] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13350.015276] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13388.179488] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13094 sec ========= 01:46:48 (1789278408) [13389.711711] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:46:50 (1789278410) === [13393.016592] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:46:53 (1789278413) === [13441.995470] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13444.868418] Key type lgssc unregistered [13445.139773] LNet: 385395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13445.148683] LNetError: 385395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13445.174276] LNet: Removed LNI 192.168.202.149@tcp [13446.113260] Key type .llcrypt unregistered [13446.115631] Key type ._llcrypt unregistered