[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491050310 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006997] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013062] *** VALIDATE selinux *** [ 0.021549] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026488] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028023] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] *** VALIDATE tmpfs *** [ 0.032429] *** VALIDATE proc *** [ 0.033239] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043021] debug: unmapping init [mem 0xffffffff97059000-0xffffffff97060fff] [ 0.045192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049024] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054286] rcu: Hierarchical SRCU implementation. [ 0.056465] smp: Bringing up secondary CPUs ... [ 0.057571] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.062130] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.179245] node 0 deferred pages initialised in 110ms [ 0.182009] devtmpfs: initialized [ 0.183389] x86/mm: Memory block size: 128MB [ 0.186000] gcov: version magic: 0x41383552 [ 0.190585] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.194106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.197402] pinctrl core: initialized pinctrl subsystem [ 0.200260] [ 0.200892] ************************************************************* [ 0.203020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205019] ** ** [ 0.208019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.211021] ** ** [ 0.213019] ** This means that this kernel is built to expose internal ** [ 0.216018] ** IOMMU data structures, which may compromise security on ** [ 0.218017] ** your system. ** [ 0.221020] ** ** [ 0.224018] ** If you see this message and you are not debugging the ** [ 0.227018] ** kernel, report this immediately to your vendor! ** [ 0.230021] ** ** [ 0.233023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236020] ************************************************************* [ 0.240064] NET: Registered protocol family 16 [ 0.241604] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.244095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.248092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253043] cpuidle: using governor menu [ 0.255213] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256804] PCI: Using configuration type 1 for base access [ 0.257162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.268080] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.271034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274195] cryptd: max_cpu_qlen set to 1000 [ 0.277341] ACPI: Added _OSI(Module Device) [ 0.279023] ACPI: Added _OSI(Processor Device) [ 0.281022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283042] ACPI: Added _OSI(Processor Aggregator Device) [ 0.288735] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.296574] ACPI: Interpreter enabled [ 0.298098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300040] ACPI: Using IOAPIC for interrupt routing [ 0.302153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.317268] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.322080] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.325035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334601] acpiphp: Slot [2] registered [ 0.336303] acpiphp: Slot [5] registered [ 0.338196] acpiphp: Slot [6] registered [ 0.340214] acpiphp: Slot [7] registered [ 0.342192] acpiphp: Slot [8] registered [ 0.343186] acpiphp: Slot [9] registered [ 0.345308] acpiphp: Slot [10] registered [ 0.347211] acpiphp: Slot [3] registered [ 0.349163] acpiphp: Slot [4] registered [ 0.351152] acpiphp: Slot [11] registered [ 0.352141] acpiphp: Slot [12] registered [ 0.354177] acpiphp: Slot [13] registered [ 0.356187] acpiphp: Slot [14] registered [ 0.357204] acpiphp: Slot [15] registered [ 0.359163] acpiphp: Slot [16] registered [ 0.361153] acpiphp: Slot [17] registered [ 0.363137] acpiphp: Slot [18] registered [ 0.364135] acpiphp: Slot [19] registered [ 0.366140] acpiphp: Slot [20] registered [ 0.369158] acpiphp: Slot [21] registered [ 0.370130] acpiphp: Slot [22] registered [ 0.372119] acpiphp: Slot [23] registered [ 0.373111] acpiphp: Slot [24] registered [ 0.374133] acpiphp: Slot [25] registered [ 0.376140] acpiphp: Slot [26] registered [ 0.378141] acpiphp: Slot [27] registered [ 0.380131] acpiphp: Slot [28] registered [ 0.382184] acpiphp: Slot [29] registered [ 0.383208] acpiphp: Slot [30] registered [ 0.385160] acpiphp: Slot [31] registered [ 0.387097] PCI host bridge to bus 0000:00 [ 0.388029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.396044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.402036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.404179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407106] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.410358] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.425951] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.431029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436747] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439877] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.443066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.446844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.466044] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.473012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.478757] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.491024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.533024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550756] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.600331] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.611027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.624020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.652733] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.687027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.699660] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.705020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.711022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.726032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.735976] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.748024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.754022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.773018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783438] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786440] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789559] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793224] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.796348] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.801184] iommu: Default domain type: Passthrough [ 0.803512] SCSI subsystem initialized [ 0.804158] ACPI: bus type USB registered [ 0.805128] usbcore: registered new interface driver usbfs [ 0.807116] usbcore: registered new interface driver hub [ 0.809090] usbcore: registered new device driver usb [ 0.810184] pps_core: LinuxPPS API ver. 1 registered [ 0.812020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.814095] PTP clock support registered [ 0.816195] EDAC MC: Ver: 3.0.0 [ 0.818235] PCI: Using ACPI for IRQ routing [ 0.820208] NetLabel: Initializing [ 0.822014] NetLabel: domain hash size = 128 [ 0.824020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826145] NetLabel: unlabeled traffic allowed by default [ 0.829217] vgaarb: loaded [ 0.834078] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.836016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.844381] clocksource: Switched to clocksource kvm-clock [ 0.958328] VFS: Disk quotas dquot_6.6.0 [ 0.960126] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963045] *** VALIDATE ramfs *** [ 0.964559] *** VALIDATE hugetlbfs *** [ 0.966225] pnp: PnP ACPI init [ 0.969139] pnp: PnP ACPI: found 6 devices [ 0.986995] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.990850] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.993144] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.995571] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.998200] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.001237] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.004850] NET: Registered protocol family 2 [ 1.007216] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.012176] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.015885] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.022688] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.026499] TCP: Hash tables configured (established 65536 bind 65536) [ 1.029982] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.033276] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.036525] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039653] NET: Registered protocol family 1 [ 1.043366] RPC: Registered named UNIX socket transport module. [ 1.045853] RPC: Registered udp transport module. [ 1.047428] RPC: Registered tcp transport module. [ 1.049155] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.051826] NET: Registered protocol family 44 [ 1.053376] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.055885] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.058278] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.061368] PCI: CLS 0 bytes, default 64 [ 1.063228] Unpacking initramfs... [ 2.535730] debug: unmapping init [mem 0xffff965a7cc54000-0xffff965a7ffbffff] [ 2.540138] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.542969] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.546362] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.060188] Initialise system trusted keyrings [ 3.062093] Key type blacklist registered [ 3.064348] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.075633] zbud: loaded [ 3.078880] *** VALIDATE nfs *** [ 3.080241] *** VALIDATE nfs4 *** [ 3.083338] pstore: using deflate compression [ 3.087069] Platform Keyring initialized [ 3.206228] NET: Registered protocol family 38 [ 3.207934] Key type asymmetric registered [ 3.209817] Asymmetric key parser 'x509' registered [ 3.211929] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.215432] io scheduler mq-deadline registered [ 3.217544] io scheduler kyber registered [ 3.219483] io scheduler bfq registered [ 3.221815] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.224936] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.228311] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.231644] ACPI: Power Button [PWRF] [ 3.237406] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.245258] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.264858] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.274653] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.296476] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.326264] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.354025] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.359634] Non-volatile memory driver v1.3 [ 3.361451] Linux agpgart interface v0.103 [ 3.399156] virtio_blk virtio1: [vda] 146752 512-byte logical blocks (75.1 MB/71.7 MiB) [ 3.401932] vda: detected capacity change from 0 to 75137024 [ 3.416614] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.419892] vdb: detected capacity change from 0 to 1073741824 [ 3.436514] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.439224] vdc: detected capacity change from 0 to 2621440000 [ 3.452828] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.459148] vdd: detected capacity change from 0 to 2621440000 [ 3.481458] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.484350] vde: detected capacity change from 0 to 4294967296 [ 3.503613] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.506130] vdf: detected capacity change from 0 to 4294967296 [ 3.517768] libphy: Fixed MDIO Bus: probed [ 3.526285] usbcore: registered new interface driver usbserial_generic [ 3.528945] usbserial: USB Serial support registered for generic [ 3.531518] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.535896] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.537950] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.540498] mousedev: PS/2 mouse device common for all mice [ 3.543968] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.546411] rtc_cmos 00:05: RTC can wake from S4 [ 3.554908] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.555409] rtc_cmos 00:05: registered as rtc0 [ 3.560272] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.561349] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.563750] intel_pstate: CPU model not supported [ 3.570769] hid: raw HID events driver (C) Jiri Kosina [ 3.573412] usbcore: registered new interface driver usbhid [ 3.575868] usbhid: USB HID core driver [ 3.577898] drop_monitor: Initializing network drop monitor service [ 3.581347] Initializing XFRM netlink socket [ 3.583676] NET: Registered protocol family 10 [ 3.587193] Segment Routing with IPv6 [ 3.588938] NET: Registered protocol family 17 [ 3.591580] mpls_gso: MPLS GSO support [ 3.597607] RAS: Correctable Errors collector initialized. [ 3.600677] AVX version of gcm_enc/dec engaged. [ 3.602956] AES CTR mode by8 optimization enabled [ 3.687734] sched_clock: Marking stable (3687707814, 0)->(4550745641, -863037827) [ 3.691989] registered taskstats version 1 [ 3.694410] Loading compiled-in X.509 certificates [ 3.696396] zswap: loaded using pool lzo/zbud [ 3.721375] Key type big_key registered [ 3.735781] Key type encrypted registered [ 3.737657] ima: No TPM chip found, activating TPM-bypass! [ 3.740181] ima: Allocated hash algorithm: sha1 [ 3.741815] ima: No architecture policies found [ 3.743973] evm: Initialising EVM extended attributes: [ 3.746299] evm: security.selinux [ 3.747539] evm: security.ima [ 3.749361] evm: security.capability [ 3.751037] evm: HMAC attrs: 0x1 [ 3.753653] rtc_cmos 00:05: setting system clock to 2026-09-08 03:36:45 UTC (1788838605) [ 3.760453] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.763315] debug: unmapping init [mem 0xffffffff96d82000-0xffffffff97058fff] [ 3.776194] Write protecting the kernel read-only data: 28672k [ 3.779338] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 3.781662] debug: unmapping init [mem 0xffffffff95d14000-0xffffffff95dfffff] [ 3.815474] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.824836] systemd[1]: Detected virtualization kvm. [ 3.826689] systemd[1]: Detected architecture x86-64. [ 3.828957] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.859736] systemd[1]: No hostname configured. [ 3.861834] systemd[1]: Set hostname to . [ 3.864340] random: systemd: uninitialized urandom read (16 bytes read) [ 3.867418] systemd[1]: Initializing machine ID from random generator. [ 4.018821] random: systemd: uninitialized urandom read (16 bytes read) [ 4.023231] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.028374] random: systemd: uninitialized urandom read (16 bytes read) [ 4.031679] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.036935] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.733189] device-mapper: uevent: version 1.0.3 [ 4.735537] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.520554] random: fast init done [ 5.578982] virtio_net virtio0 ens2: renamed from eth0 [ 5.658560] scsi host0: ata_piix [ 5.720436] scsi host1: ata_piix [ 5.722186] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.725051] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.076073] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.256080] random: crng init done [ 10.257359] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.807786] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.092646] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.390180] SELinux: Disabled at runtime. [ 12.455905] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.465718] systemd[1]: Detected virtualization kvm. [ 12.467803] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.006963] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.011275] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.016891] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.022338] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.027622] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.037923] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.046208] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] [ 13.146055] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.593624] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.899481] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.949298] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.105773] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.122537] EDAC sbridge: Ver: 1.1.2 [ 15.835169] Key type dns_resolver registered [ 16.147604] NFS: Registering the id_resolver key type [ 16.150235] Key type id_resolver registered [ 16.151936] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 47.528380] hrtimer: interrupt took 5278796 ns [ 53.491042] libcfs: loading out-of-tree module taints kernel. [ 53.600783] Key type ._llcrypt registered [ 53.604993] Key type .llcrypt registered [ 53.764709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 74.313016] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 75.787488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.797537] alg: No test for adler32 (adler32-zlib) [ 77.044798] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 77.773291] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 79.472747] Key type lgssc registered [ 81.637223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.428908] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 143.109838] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 156.641243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.693973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.934606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.963623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.053040] Lustre: lustre-MDT0000: new disk, initializing [ 158.131341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.155728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.871799] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.036970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.168314] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.231206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.243656] Lustre: Skipped 1 previous similar message [ 176.368713] Lustre: lustre-MDT0001: new disk, initializing [ 176.455132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.499463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.522061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 180.926658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.901944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.247121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.520709] Lustre: lustre-OST0000: new disk, initializing [ 197.524295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.530690] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.586461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.369289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.390595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.510676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.336384] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.497567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.644244] Lustre: lustre-OST0001: new disk, initializing [ 220.648782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.655498] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.721484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.228505] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.900279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.910122] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.965358] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.663749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.603343] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.851882] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.746905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 265.093668] Lustre: DEBUG MARKER: Client: 2.17.57.117 [ 268.078734] Lustre: DEBUG MARKER: MDS: 2.17.57.117 [ 271.227257] Lustre: DEBUG MARKER: OSS: 2.17.57.117 [ 273.156638] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 23:41:13 EDT 2026 [ 291.092993] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 292.935223] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.739629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.749440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.775772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.756267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.770658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.778707] Lustre: Skipped 1 previous similar message [ 310.783849] Lustre: Skipped 1 previous similar message [ 315.471294] Lustre: server umount lustre-MDT0000 complete [ 319.289647] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838921 with bad export cookie 5830851094965634814 [ 319.301026] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.306383] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.994920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.013982] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.014649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.014654] Lustre: Skipped 2 previous similar messages [ 321.015747] Lustre: Skipped 2 previous similar messages [ 321.043564] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 325.024127] LustreError: 6511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.024268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.047477] Lustre: Skipped 2 previous similar messages [ 325.710234] Lustre: server umount lustre-MDT0001 complete [ 330.454529] Lustre: server umount lustre-OST0000 complete [ 334.773485] Lustre: server umount lustre-OST0001 complete [ 343.388563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 353.623155] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 394.368414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.634771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.657369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.720803] Lustre: lustre-MDT0000: new disk, initializing [ 394.783651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.807229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.871594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 412.134541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.384431] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.499496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.504500] Lustre: Skipped 1 previous similar message [ 412.583564] Lustre: lustre-MDT0001: new disk, initializing [ 412.731121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.766788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.782033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.096219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 428.718594] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.342568] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.714455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.103751] Lustre: lustre-OST0000: new disk, initializing [ 443.114621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.123117] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.254525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.255579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.268763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.413607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.707057] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 459.640960] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.288116] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.485836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.830311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.148992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.130874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.145445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.165607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.874691] Lustre: server umount lustre-OST0000 complete [ 486.368969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.386091] Lustre: Skipped 2 previous similar messages [ 486.387523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.401046] Lustre: Skipped 2 previous similar messages [ 491.502259] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.518465] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 491.666348] Lustre: server umount lustre-MDT0000 complete [ 495.584908] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.600501] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 496.150619] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839097 with bad export cookie 5830851094965636354 [ 496.164221] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.168731] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 496.692561] Lustre: server umount lustre-MDT0001 complete [ 503.659303] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:45:03 (1788839103) [ 524.487466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.970888] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.046031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.956984] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 534.505218] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.527547] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 539.322210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.861712] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 551.564180] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.421627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 559.840792] Lustre: Failing over lustre-MDT0000 [ 560.096157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.098395] Lustre: server umount lustre-MDT0000 complete [ 560.100293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.103246] LustreError: 19909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.265565] Lustre: 3637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839161/real 1788839161] req@ffff965afe1bdf80 x1875733306830208/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788839177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 575.292283] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.656243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.703090] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x50eb582d7d9fd288 [ 585.722664] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 591.138505] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 591.330669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 591.342432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 591.373178] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 602.980741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 609.768577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.775155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.791582] Lustre: Skipped 2 previous similar messages [ 609.809858] Lustre: Skipped 1 previous similar message [ 614.178463] Lustre: server umount lustre-MDT0000 complete [ 614.901085] LustreError: 19909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.917445] LustreError: 19909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 619.643456] LustreError: 19894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839221 with bad export cookie 5830851094965637768 [ 619.653311] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.664421] LustreError: 19894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 620.339573] Lustre: server umount lustre-MDT0001 complete [ 633.225491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.800170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.816130] Lustre: Skipped 2 previous similar messages [ 639.923068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 650.789954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.898990] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 664.813504] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 672.896369] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 675.748064] Lustre: Failing over lustre-MDT0000 [ 676.037738] Lustre: server umount lustre-MDT0000 complete [ 676.832505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.842231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.935955] LustreError: 23252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.957464] LustreError: 23252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 696.205516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.291623] Lustre: 3636:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839282/real 1788839282] req@ffff9659c7ce8000 x1875733306861696/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788839298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 696.313305] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.323783] LustreError: 3634:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9659c70a1180 x1875733306863488/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 696.750275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 701.411978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 701.928508] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 701.930490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 701.971636] Lustre: Skipped 1 previous similar message [ 701.993259] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 711.112406] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 717.287519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 717.291955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.314673] Lustre: Skipped 1 previous similar message [ 721.453961] Lustre: server umount lustre-MDT0000 complete [ 725.698925] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839327 with bad export cookie 5830851094965638706 [ 725.700265] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.711429] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 726.037540] Lustre: server umount lustre-MDT0001 complete [ 735.126976] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:48:55 (1788839335) [ 752.865516] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 762.898805] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 811.913460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 827.529757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.768129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 827.796315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 827.865781] Lustre: lustre-MDT0000: new disk, initializing [ 827.932645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.938338] Lustre: Skipped 2 previous similar messages [ 827.951595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.256743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 844.535122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.669725] Lustre: 29629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 845.073584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 850.405529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 860.159387] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 867.060934] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 873.730335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.160228] Lustre: 31601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 875.552442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 875.631263] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 875.686373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 875.706887] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 881.189739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 892.704697] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.006739] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 927.214727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.217954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.234266] LustreError: Skipped 1 previous similar message [ 927.236350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.236359] Lustre: Skipped 2 previous similar messages [ 927.286677] Lustre: Skipped 4 previous similar messages [ 930.137552] Lustre: server umount lustre-OST0000 complete [ 941.137268] Lustre: server umount lustre-MDT0000 complete [ 942.566107] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.597617] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 946.169171] LustreError: 30426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839547 with bad export cookie 5830851094965639112 [ 946.177864] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 946.185333] LustreError: 30426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 946.664702] Lustre: server umount lustre-MDT0001 complete [ 960.102680] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 963.550061] Key type lgssc unregistered [ 963.961815] LNet: 33581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.972617] LNetError: 33581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.989424] LNet: Removed LNI 192.168.204.146@tcp [ 965.353165] Key type .llcrypt unregistered [ 965.354692] Key type ._llcrypt unregistered [ 995.803851] Key type ._llcrypt registered [ 995.808871] Key type .llcrypt registered [ 995.982871] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 997.720195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.750874] alg: No test for adler32 (adler32-zlib) [ 998.826505] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 999.095147] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1000.815171] Key type lgssc registered [ 1002.165683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1011.328605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.392863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.904596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.341875] LustreError: 34991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.358377] LustreError: 34991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1018.629105] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1022.441705] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.556398] LustreError: 34991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.486906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.026484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.529382] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1043.837505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.238694] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.941384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.365572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.580563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1073.706839] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1074.123082] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1082.961284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1100.665987] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:55:00 (1788839700) [ 1102.784875] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1104.744197] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1112.248196] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:55:12 (1788839712) [ 1120.547578] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:55:20 (1788839720) [ 1121.744231] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1130.809996] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1130.817213] Lustre: Skipped 2 previous similar messages [ 1132.939652] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:55:32 (1788839732) [ 1135.110941] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1135.117745] Lustre: Skipped 2 previous similar messages [ 1143.974681] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1143.980292] Lustre: Skipped 4 previous similar messages [ 1159.726357] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:55:59 (1788839759) [ 1163.247128] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1167.144437] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1177.533566] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:56:17 (1788839777) [ 1180.105497] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1180.110956] Lustre: Skipped 6 previous similar messages [ 1196.703939] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:56:36 (1788839796) [ 1215.971767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.973410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.994060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.007874] Lustre: Skipped 1 previous similar message [ 1218.348605] Lustre: server umount lustre-OST0000 complete [ 1223.649057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1223.658986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.678750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.684156] Lustre: Skipped 1 previous similar message [ 1226.218212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.230073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.064843] Lustre: server umount lustre-MDT0000 complete [ 1231.332563] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.359022] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1232.736370] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839834 with bad export cookie 18287846247907924676 [ 1232.745493] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.749683] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1233.173649] Lustre: server umount lustre-MDT0001 complete [ 1244.539720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1248.000866] Key type lgssc unregistered [ 1248.287492] LNet: 42868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.299355] LNetError: 42868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.319827] LNet: Removed LNI 192.168.204.146@tcp [ 1249.324817] Key type .llcrypt unregistered [ 1249.327159] Key type ._llcrypt unregistered [ 1272.100143] Key type ._llcrypt registered [ 1272.105431] Key type .llcrypt registered [ 1272.246678] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1288.448788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1289.780821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.897452] alg: No test for adler32 (adler32-zlib) [ 1291.149950] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1291.576074] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1293.401599] Key type lgssc registered [ 1294.645440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.026346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.088486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.383805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1332.454272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1332.622734] Lustre: lustre-MDT0000: new disk, initializing [ 1332.715708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.737172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1337.453775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1348.904187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.051303] Lustre: 46018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1349.132648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1349.143082] Lustre: Skipped 1 previous similar message [ 1349.241821] Lustre: lustre-MDT0001: new disk, initializing [ 1349.383024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.453284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1349.465375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1355.544534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1366.603843] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.235707] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1380.495732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.769476] Lustre: lustre-OST0000: new disk, initializing [ 1380.772826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1380.780704] Lustre: 47989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1380.852615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.064778] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1389.162649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1389.174327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1389.310701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1399.321966] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1404.674133] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1404.923885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1409.974583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.256844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1415.148600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.155423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1415.175607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.861985] Lustre: server umount lustre-OST0000 complete [ 1425.379941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.387877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.394441] Lustre: Skipped 2 previous similar messages [ 1425.416278] Lustre: Skipped 2 previous similar messages [ 1430.498331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.508692] Lustre: Skipped 1 previous similar message [ 1435.618463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.628513] Lustre: Skipped 1 previous similar message [ 1437.151167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1437.471029] Lustre: server umount lustre-MDT0000 complete [ 1441.496473] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840043 with bad export cookie 10386920406024114586 [ 1441.508392] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.510683] LustreError: 46010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1442.247569] Lustre: server umount lustre-MDT0001 complete [ 1452.567985] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:00:52 (1788840052) [ 1459.442980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.124911] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.151628] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1460.280546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.075503] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1465.315827] LustreError: 50452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.479757] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.034389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.501696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.341630] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1488.694159] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.043829] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1502.991466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.444660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.640803] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1517.813441] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.748081] LustreError: 52993:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1526.629560] Lustre: Modifying parameter general.timeout=1 in log params [ 1528.801870] Lustre: Modifying parameter general.timeout=3 in log params [ 1528.820163] Lustre: Skipped 1 previous similar message [ 1530.984935] Lustre: Modifying parameter general.timeout=5 in log params [ 1530.993244] Lustre: Skipped 1 previous similar message [ 1535.946083] Lustre: Modifying parameter general.timeout=9 in log params [ 1535.956414] Lustre: Skipped 3 previous similar messages [ 1544.201642] Lustre: Modifying parameter general.timeout=17 in log params [ 1544.207517] Lustre: Skipped 7 previous similar messages [ 1561.025657] Lustre: Modifying parameter general.timeout=31 in log params [ 1561.039349] Lustre: Skipped 13 previous similar messages [ 1594.037278] Lustre: Modifying parameter general.timeout=62 in log params [ 1594.050781] Lustre: Skipped 30 previous similar messages [ 1658.432776] Lustre: Modifying parameter general.timeout=124 in log params [ 1658.436073] Lustre: Skipped 61 previous similar messages [ 1675.541666] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:04:35 (1788840275) [ 1678.565360] Lustre: 59956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1680.577470] Lustre: 60054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1684.304742] Lustre: 60267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1684.312655] Lustre: 60267:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1717.652342] Lustre: 61829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1717.668044] Lustre: 61829:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1717.701528] LustreError: 61829:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1721.929666] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:05:22 (1788840322) [ 1735.652845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1735.655199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.663301] LustreError: Skipped 1 previous similar message [ 1735.674155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1735.689752] Lustre: Skipped 2 previous similar messages [ 1735.700319] Lustre: Skipped 1 previous similar message [ 1740.475403] Lustre: server umount lustre-MDT0000 complete [ 1740.769737] LustreError: 50452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.802165] LustreError: 50452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1744.278171] LustreError: 53042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840346 with bad export cookie 10386920406024115468 [ 1744.287983] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.300352] LustreError: 53042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1744.764795] Lustre: server umount lustre-MDT0001 complete [ 1760.265314] Lustre: server umount lustre-OST0000 complete [ 1795.046319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.618963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1795.651362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1795.663460] Lustre: Skipped 28 previous similar messages [ 1796.101546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.393780] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1814.781287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.025317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1815.081403] Lustre: 64361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1822.459650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1833.654922] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1840.652452] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.682872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1850.054983] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1850.494988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.507744] Lustre: Skipped 1 previous similar message [ 1857.940775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1867.480344] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.659991] Lustre: 66929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1881.568773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1881.577463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.597524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.236071] Lustre: server umount lustre-OST0000 complete [ 1892.321445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.323195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.344948] Lustre: Skipped 2 previous similar messages [ 1895.187027] Lustre: server umount lustre-MDT0000 complete [ 1897.440228] LustreError: 64371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.471232] LustreError: 64371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1899.054045] LustreError: 66931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840500 with bad export cookie 10386920406024118723 [ 1899.056457] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.063937] LustreError: 66931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1899.433977] Lustre: server umount lustre-MDT0001 complete [ 1911.933496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1915.207171] Key type lgssc unregistered [ 1915.483329] LNet: 68141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.490516] LNetError: 68141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.509865] LNet: Removed LNI 192.168.204.146@tcp [ 1916.688475] Key type .llcrypt unregistered [ 1916.692651] Key type ._llcrypt unregistered [ 1946.775499] Key type ._llcrypt registered [ 1946.787255] Key type .llcrypt registered [ 1946.939471] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1948.519694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.587359] alg: No test for adler32 (adler32-zlib) [ 1949.897195] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1950.286391] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1952.112366] Key type lgssc registered [ 1953.736490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.586079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.613882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.373136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.777445] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1975.781033] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.802424] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1980.896711] LustreError: 69560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.947813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.805427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.469985] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1996.767585] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.126492] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2012.558952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2013.191496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.600345] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2029.177651] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2048.597231] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2061.283461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.300251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.311373] Lustre: Skipped 1 previous similar message [ 2061.341497] Lustre: Skipped 2 previous similar messages [ 2065.286926] Lustre: server umount lustre-MDT0000 complete [ 2066.405120] LustreError: 69556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.166418] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840670 with bad export cookie 13932387039806311848 [ 2069.176738] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.193183] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2069.533496] Lustre: server umount lustre-MDT0001 complete [ 2085.083381] Lustre: server umount lustre-OST0000 complete [ 2113.860030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.299205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2114.343959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2114.603271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.468465] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2130.109841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.228197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2130.268634] Lustre: 74459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2130.826370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.907561] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2145.068279] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.904524] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2161.861592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2162.091254] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2162.503972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.000734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2180.553754] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.944703] Lustre: 77033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2185.129634] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2191.333832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2191.345826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.358485] Lustre: Skipped 1 previous similar message [ 2191.366447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2196.451412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2196.460724] Lustre: Skipped 2 previous similar messages [ 2197.592388] Lustre: server umount lustre-OST0000 complete [ 2206.692409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.705972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.712912] Lustre: Skipped 2 previous similar messages [ 2206.729387] Lustre: Skipped 1 previous similar message [ 2208.776938] Lustre: server umount lustre-MDT0000 complete [ 2211.818725] LustreError: 74464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.844335] LustreError: 74464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2213.153924] LustreError: 74451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840814 with bad export cookie 13932387039806312695 [ 2213.160078] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.162995] LustreError: 74451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2213.670840] Lustre: server umount lustre-MDT0001 complete [ 2228.117280] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2231.087494] Key type lgssc unregistered [ 2231.508647] LNet: 78246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.513901] LNetError: 78246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.529839] LNet: Removed LNI 192.168.204.146@tcp [ 2232.531716] Key type .llcrypt unregistered [ 2232.535898] Key type ._llcrypt unregistered [ 2262.029984] Key type ._llcrypt registered [ 2262.034216] Key type .llcrypt registered [ 2262.151236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2263.598711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.680349] alg: No test for adler32 (adler32-zlib) [ 2264.831355] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2265.135650] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2266.815198] Key type lgssc registered [ 2268.083398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2277.196062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.230709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.802620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.887249] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2289.121839] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.145957] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2293.501459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.054900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2299.763312] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2309.301957] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2317.742371] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.951132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.753404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.126636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2342.866997] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.611540] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2355.596815] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.366536] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.371120] Lustre: Skipped 1 previous similar message [ 2359.629530] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2359.635803] Lustre: Skipped 1 previous similar message [ 2363.715219] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2363.723845] Lustre: Skipped 3 previous similar messages [ 2371.851360] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2371.856368] Lustre: Skipped 7 previous similar messages [ 2388.675269] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2388.696109] Lustre: Skipped 16 previous similar messages [ 2421.081506] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2421.093692] Lustre: Skipped 31 previous similar messages [ 2463.558977] Lustre: 87762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2467.312944] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:17:46 (1788841066) [ 2469.391665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2471.557215] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:17:51 (1788841071) [ 2481.657603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2481.660312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.678036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.699444] Lustre: Skipped 1 previous similar message [ 2486.771299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.786154] Lustre: Skipped 2 previous similar messages [ 2491.874307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.891878] Lustre: Skipped 1 previous similar message [ 2492.383180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2492.581332] Lustre: server umount lustre-OST0000 complete [ 2498.528093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.547184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.570972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.070348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.615201] Lustre: server umount lustre-MDT0000 complete [ 2507.233162] LustreError: 82959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.257421] LustreError: 82959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2508.353815] LustreError: 82247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841110 with bad export cookie 9429296868064401433 [ 2508.356035] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.377637] LustreError: 82247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2508.900562] Lustre: server umount lustre-MDT0001 complete [ 2520.726976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2523.546265] Key type lgssc unregistered [ 2523.857957] LNet: 89012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.868607] LNetError: 89012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.887367] LNet: Removed LNI 192.168.204.146@tcp [ 2524.754683] Key type .llcrypt unregistered [ 2524.757285] Key type ._llcrypt unregistered [ 2529.913059] Key type ._llcrypt registered [ 2529.918036] Key type .llcrypt registered [ 2530.011370] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2534.809167] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 2536.178615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.247561] alg: No test for adler32 (adler32-zlib) [ 2537.612701] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2537.862230] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2539.561288] Key type lgssc registered [ 2541.116835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2542.501386] LustreError: 90207:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2562.293469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2564.783280] LustreError: 90207:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2571.904979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2571.917500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.569132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.144842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2584.036695] LustreError: 91223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.068827] LustreError: 91223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2588.435817] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:19:48 (1788841188) [ 2589.156131] LustreError: 91224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.334693] LustreError: 91255:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2593.347638] LustreError: 91255:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2593.359708] LustreError: 91255:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2593.927056] Lustre: server umount lustre-MDT0000 complete [ 2609.312394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2612.567356] Key type lgssc unregistered [ 2612.894045] LNet: 92747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.906229] LNetError: 92747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.926502] LNet: Removed LNI 192.168.204.146@tcp [ 2613.894300] Key type .llcrypt unregistered [ 2613.899610] Key type ._llcrypt unregistered [ 2636.129155] Key type ._llcrypt registered [ 2636.134141] Key type .llcrypt registered [ 2636.297182] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2637.431058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2637.619793] alg: No test for adler32 (adler32-zlib) [ 2638.988201] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2639.274906] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2641.023217] Key type lgssc registered [ 2642.490974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2651.475971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2651.500976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.304990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.547268] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2662.892364] LustreError: 93962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.928996] LustreError: 93962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2668.003698] LustreError: 93963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.453973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.080256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.227786] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2689.526157] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2699.434912] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2707.140537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.965610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.066944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2723.613764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2776.511314] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:22:56 (1788841376) [ 2778.529155] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2780.595596] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:23:00 (1788841380) [ 2788.834150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.837994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.853238] Lustre: Skipped 1 previous similar message [ 2788.867324] Lustre: Skipped 2 previous similar messages [ 2791.842469] Lustre: server umount lustre-MDT0000 complete [ 2793.956636] LustreError: 94696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.975142] LustreError: 94696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2795.839850] LustreError: 93943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841397 with bad export cookie 2241937467003232807 [ 2795.859814] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.454748] Lustre: server umount lustre-MDT0001 complete [ 2811.404355] Lustre: server umount lustre-OST0000 complete [ 2825.210831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.684988] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.867439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.331687] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.836917] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2840.555943] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.582263] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2841.869256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.720468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.323662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2855.289328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2860.264924] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2873.889901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.192055] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2874.207109] LustreError: 98252:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2874.251504] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2874.266263] LustreError: 100347:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2874.272473] LustreError: 100347:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2874.285413] LustreError: 100347:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2874.451664] Lustre: server umount lustre-OST0000 complete [ 2874.464261] LustreError: 100347:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2883.806510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.954076] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2883.961741] LustreError: 98251:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2883.979427] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2883.988298] LustreError: 100812:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2883.993981] LustreError: 100812:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2883.998904] LustreError: 100812:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2884.071402] Lustre: server umount lustre-OST0000 complete [ 2884.075498] LustreError: 100812:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2895.846857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.210738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2896.222087] Lustre: Found index 0 for lustre-OST0000, updating log [ 2896.231031] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2896.421046] Lustre: lustre-OST0000: new disk, initializing [ 2896.426311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2896.443085] Lustre: 101347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2896.871859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.620658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2909.384781] Lustre: Failing over lustre-OST0000 [ 2909.614745] Lustre: server umount lustre-OST0000 complete [ 2916.837793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.845446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.867848] Lustre: Skipped 2 previous similar messages [ 2921.957838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.966244] Lustre: Skipped 2 previous similar messages [ 2925.160921] Lustre: server umount lustre-MDT0000 complete [ 2927.072462] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.105813] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2928.751173] LustreError: 98244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841530 with bad export cookie 2241937467003235551 [ 2928.766485] LustreError: 98244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2928.778108] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.103608] Lustre: server umount lustre-MDT0001 complete [ 2939.157388] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:25:39 (1788841539) [ 2954.789882] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2965.744438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.329954] LustreError: 103693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.409423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.670928] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2979.349631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.227824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2987.292392] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2994.326611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.177677] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3001.374820] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3004.409839] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3004.418165] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3004.434833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3009.511536] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 3009.520605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3010.181469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.266772] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3010.381162] Lustre: lustre-OST0001: new disk, initializing [ 3010.388726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3010.398873] Lustre: 106012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.656848] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 3010.680163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3010.693830] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3010.702103] Lustre: Skipped 2 previous similar messages [ 3011.714390] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 3011.716071] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3011.732195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3017.550426] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3020.435963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3020.445242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3020.558501] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3025.677130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3031.850433] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3036.142028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.149894] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3036.154805] Lustre: Skipped 1 previous similar message [ 3040.855201] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 3040.864855] Lustre: Skipped 2 previous similar messages [ 3042.132380] Lustre: server umount lustre-MDT0001 complete [ 3045.941435] LustreError: 104748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.962379] LustreError: 104748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3050.729169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.818280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3050.823574] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3050.865548] Lustre: 103682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3051.052946] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (not set up) [ 3051.062120] Lustre: Skipped 3 previous similar messages [ 3052.551949] LustreError: 107583:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3052.561813] LustreError: 107583:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3052.581208] LustreError: 103685:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3052.694304] LustreError: 107595:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 3052.703855] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3052.703855] [ 3055.740530] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3056.633625] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3065.303531] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:27:45 (1788841665) [ 3086.039169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3104.723246] Lustre: 109626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3128.965329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3132.744640] Lustre: 110760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3225.567837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.573055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.582716] Lustre: Skipped 3 previous similar messages [ 3225.602750] Lustre: Skipped 3 previous similar messages [ 3229.968636] Lustre: server umount lustre-MDT0000 complete [ 3230.694203] LustreError: 107214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.708993] LustreError: 107214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3233.903056] LustreError: 103675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841835 with bad export cookie 2241937467003236552 [ 3233.904218] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3233.914775] LustreError: 103675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3234.244115] Lustre: server umount lustre-MDT0001 complete [ 3247.540382] Lustre: server umount lustre-OST0000 complete [ 3251.677363] Lustre: 93273:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841837/real 1788841837] req@ffff9659d24d0000 x1875735993484544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841853 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3262.119910] Lustre: server umount lustre-OST0001 complete [ 3268.026935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3279.433828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3288.470231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3299.054158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3321.057890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3334.958223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.987187] Lustre: lustre-MDT0000: reset Object Index mappings [ 3335.901879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.919857] Lustre: Skipped 1 previous similar message [ 3341.425944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3352.575495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.588263] Lustre: lustre-MDT0001: reset Object Index mappings [ 3359.319305] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3362.711288] Lustre: 115864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3369.543732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.466845] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3377.876784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3380.569572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3387.071826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.023653] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3393.029873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3396.229159] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3406.058221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3411.226039] Lustre: 117744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3467.525665] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:34:27 (1788842067) [ 3495.402597] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.406536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.415106] Lustre: Skipped 4 previous similar messages [ 3495.429257] Lustre: Skipped 3 previous similar messages [ 3500.007452] Lustre: server umount lustre-MDT0000 complete [ 3500.521031] LustreError: 115453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.551424] LustreError: 115453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3504.394940] LustreError: 114696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842106 with bad export cookie 2241937467003283669 [ 3504.405069] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.641283] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.288597] Lustre: server umount lustre-MDT0001 complete [ 3522.538743] Lustre: server umount lustre-OST0000 complete [ 3533.091753] Lustre: server umount lustre-OST0001 complete [ 3540.419414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3549.746709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3590.377565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.575331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.594048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3590.644319] Lustre: lustre-MDT0000: new disk, initializing [ 3590.698960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.706485] Lustre: Skipped 3 previous similar messages [ 3590.720947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3595.269877] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3602.396991] Lustre: server umount lustre-MDT0000 complete [ 3612.558615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.849555] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3612.869550] Lustre: 122009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3612.889945] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3617.522887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3623.622367] Lustre: server umount lustre-MDT0000 complete [ 3635.668239] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:37:15 (1788842235) [ 3638.602873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3641.014958] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:37:20 (1788842240) [ 3663.215077] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3673.406035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.790933] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3687.698703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.837966] Lustre: 124060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3687.922140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3687.928604] Lustre: Skipped 1 previous similar message [ 3688.001409] Lustre: lustre-MDT0001: new disk, initializing [ 3688.127412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3688.146249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3693.532443] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3698.856480] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3705.420053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.656348] Lustre: lustre-OST0000: new disk, initializing [ 3705.660403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3705.669439] Lustre: 125639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3707.474834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3707.487074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3707.579494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3712.824062] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3725.088946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3725.241626] Lustre: lustre-OST0001: new disk, initializing [ 3725.248965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3725.253733] Lustre: 126511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3727.025161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3727.041369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3727.137863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3733.087895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3745.176421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3749.214851] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4176.615597] Lustre: 132937:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4176.628892] Lustre: 132937:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4176.637410] Lustre: 132937:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4176.647289] Lustre: 132937:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4176.652519] Lustre: 132937:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4176.661530] Lustre: 132937:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4473.011363] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:51:11 (1788843071) [ 4490.218062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.241896] Lustre: Skipped 3 previous similar messages [ 4490.243992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4490.253696] Lustre: Skipped 4 previous similar messages [ 4496.483973] Lustre: server umount lustre-MDT0000 complete [ 4500.451454] LustreError: 133007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843102 with bad export cookie 2241937467003353312 [ 4500.455595] LustreError: 126166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.465971] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.491309] LustreError: 133007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4500.549886] LustreError: 126166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4505.570136] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.576419] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4505.582540] Lustre: Skipped 2 previous similar messages [ 4505.591966] Lustre: Skipped 7 previous similar messages [ 4507.453123] Lustre: server umount lustre-MDT0001 complete [ 4518.199895] Lustre: server umount lustre-OST0000 complete [ 4528.498202] Lustre: server umount lustre-OST0001 complete [ 4535.177105] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4547.481969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4589.626247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.181930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4590.221026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4590.369805] Lustre: lustre-MDT0000: new disk, initializing [ 4590.488179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.495575] Lustre: Skipped 5 previous similar messages [ 4590.525512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4596.144212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4607.932623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.048560] Lustre: 135996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4608.116404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4608.122750] Lustre: Skipped 1 previous similar message [ 4608.223307] Lustre: lustre-MDT0001: new disk, initializing [ 4608.437516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4608.455136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4615.042267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4627.217800] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4634.409804] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4642.643506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.000734] Lustre: lustre-OST0000: new disk, initializing [ 4643.007913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4643.018751] Lustre: 137966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4643.161467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.166545] Lustre: Skipped 1 previous similar message [ 4644.277084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4644.288736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4644.408143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4650.281788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4661.166078] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4673.194284] Lustre: lustre-MDD0000: changelog on [ 4676.537723] Lustre: lustre-MDD0001: changelog on [ 6839.719634] Lustre: lustre-MDD0001: changelog off [ 6842.124590] Lustre: 136004:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6843.054837] Lustre: lustre-MDD0000: changelog off [ 6851.041031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6851.055153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.064295] LustreError: Skipped 2 previous similar messages [ 6851.071699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6851.085393] Lustre: Skipped 1 previous similar message [ 6851.108265] Lustre: Skipped 2 previous similar messages [ 6856.165674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6856.181094] Lustre: Skipped 1 previous similar message [ 6856.434402] Lustre: server umount lustre-OST0000 complete [ 6861.282439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6861.283156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.333935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6861.341884] Lustre: Skipped 1 previous similar message [ 6871.525709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6871.534028] Lustre: Skipped 3 previous similar messages [ 6875.103160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6875.513647] Lustre: server umount lustre-MDT0000 complete [ 6876.654567] LustreError: 136869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.716744] LustreError: 136869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6879.882497] LustreError: 136792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845481 with bad export cookie 2241937467004135457 [ 6879.890087] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.899375] LustreError: 136792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6880.479691] Lustre: server umount lustre-MDT0001 complete [ 6894.107878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6897.681862] Key type lgssc unregistered [ 6898.120664] LNet: 147147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.130665] LNetError: 147147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6898.156164] LNet: Removed LNI 192.168.204.146@tcp [ 6899.237161] Key type .llcrypt unregistered [ 6899.242493] Key type ._llcrypt unregistered [ 6921.030926] Key type ._llcrypt registered [ 6921.037336] Key type .llcrypt registered [ 6921.144454] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6935.924407] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6936.824592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6936.843357] alg: No test for adler32 (adler32-zlib) [ 6937.999490] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 6938.232416] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6939.879209] Key type lgssc registered [ 6940.928111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6976.092525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6976.122161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.390616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6977.446863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6977.556423] Lustre: lustre-MDT0000: new disk, initializing [ 6977.624276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.647129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6982.521626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6992.839026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.000932] Lustre: 150348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6993.123552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6993.134960] Lustre: Skipped 1 previous similar message [ 6993.251439] Lustre: lustre-MDT0001: new disk, initializing [ 6993.357493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.402125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6993.413443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6997.806416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7009.104438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7015.964928] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7023.452694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.850730] Lustre: lustre-OST0000: new disk, initializing [ 7023.854785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7023.873435] Lustre: 152320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7024.022347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.359718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7030.372717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7030.525216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7031.426852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7042.883509] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.498120] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 7052.564813] LustreError: 153324:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7052.574405] LustreError: 153324:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7054.204158] LustreError: 153421:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7066.596840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.599177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7066.621281] Lustre: Skipped 2 previous similar messages [ 7066.636089] Lustre: Skipped 2 previous similar messages [ 7071.723275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.734055] Lustre: Skipped 2 previous similar messages [ 7072.568541] Lustre: server umount lustre-MDT0000 complete [ 7076.053883] LustreError: 150339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845677 with bad export cookie 8725359991088493129 [ 7076.064861] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.073479] LustreError: 150339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7076.484835] Lustre: server umount lustre-MDT0001 complete [ 7091.175645] Lustre: server umount lustre-OST0000 complete [ 7114.217948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7127.310241] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7173.442151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.707510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7173.734319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7173.844972] Lustre: lustre-MDT0000: new disk, initializing [ 7173.991581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.020656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7179.144631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7191.146862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.330771] Lustre: 157138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7191.399728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7191.411369] Lustre: Skipped 1 previous similar message [ 7191.515867] Lustre: lustre-MDT0001: new disk, initializing [ 7191.610907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7191.652273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7191.671328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7197.024311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7207.298303] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7214.627190] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7223.287096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.650524] Lustre: lustre-OST0000: new disk, initializing [ 7223.653918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7223.658745] Lustre: 159107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7223.775356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7231.299603] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7234.130526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7234.149321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7234.291110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7242.412177] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7248.526624] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7248.789522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7254.729803] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7254.984220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7260.130165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.142068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7260.150422] Lustre: Skipped 1 previous similar message [ 7260.150820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.167312] LustreError: Skipped 1 previous similar message [ 7260.194582] Lustre: Skipped 1 previous similar message [ 7263.563889] Lustre: server umount lustre-OST0000 complete [ 7268.322542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7268.328539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.346967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.374791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.582128] Lustre: server umount lustre-MDT0000 complete [ 7277.650426] LustreError: 157934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845879 with bad export cookie 8725359991088494095 [ 7277.653914] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7277.667310] LustreError: 157934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7278.074698] Lustre: server umount lustre-MDT0001 complete [ 7282.581913] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:38:01 (1788845881) [ 7284.649934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7286.906441] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:38:06 (1788845886) [ 7296.818140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.271414] LustreError: 161523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.302909] LustreError: 161523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7297.362662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.896170] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7302.631451] LustreError: 161524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.747866] LustreError: 161523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.184086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.674610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7325.170731] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.128918] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7340.523772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.010173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.027130] Lustre: Skipped 1 previous similar message [ 7349.672351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7359.190427] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.642429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.678060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.688482] Lustre: Skipped 4 previous similar messages [ 7371.249379] Lustre: server umount lustre-MDT0000 complete [ 7371.746429] LustreError: 162765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.772655] LustreError: 162765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7375.306552] LustreError: 161505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845977 with bad export cookie 8725359991088494977 [ 7375.319915] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.329055] LustreError: 161505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7375.441896] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 7375.447201] Lustre: Skipped 1 previous similar message [ 7376.873976] LustreError: 161520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.882590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.889300] LustreError: 161520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7376.917811] Lustre: Skipped 2 previous similar messages [ 7381.845268] Lustre: server umount lustre-MDT0001 complete [ 7388.526115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7402.555413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.801888] LustreError: 165368:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7402.811991] Lustre: 165368:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7405.536415] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.556498] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7426.016518] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.040952] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7461.419929] LustreError: 165374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7461.437520] LustreError: 165374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 7463.391662] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7463.400797] Lustre: 165368:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7504.371876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7508.981203] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7509.481622] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7518.533091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.671154] LustreError: 166112:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7520.689656] LustreError: 166112:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000c8926874 x1875740500032000/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788846122 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7520.719559] LustreError: 166112:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7520.810897] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (not set up) [ 7520.815269] Lustre: Skipped 3 previous similar messages [ 7525.653590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7526.394309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7534.144650] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7535.789455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7541.642647] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7543.275819] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7565.146157] Lustre: server umount lustre-MDT0000 complete [ 7567.334675] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7567.340207] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.346286] Lustre: Skipped 1 previous similar message [ 7567.346534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7567.364461] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7568.701099] LustreError: 161505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846170 with bad export cookie 8725359991088501431 [ 7568.704178] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7568.711134] LustreError: 161505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7568.984105] Lustre: server umount lustre-MDT0001 complete [ 7584.056576] Lustre: server umount lustre-OST0000 complete [ 7609.206749] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7622.899503] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7667.000108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7667.236311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7667.257651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7667.352241] Lustre: lustre-MDT0000: new disk, initializing [ 7667.524051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.535482] Lustre: Skipped 1 previous similar message [ 7667.582347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7673.559386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7685.982161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.137102] Lustre: 170472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7686.446993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7691.880190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7702.734816] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.458974] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.979149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.311756] Lustre: lustre-OST0000: new disk, initializing [ 7717.319266] Lustre: Skipped 1 previous similar message [ 7717.328652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7717.337572] Lustre: Skipped 2 previous similar messages [ 7717.347844] Lustre: 172442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7724.659625] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7725.635111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7725.649607] Lustre: Skipped 1 previous similar message [ 7725.691041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7725.851951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7735.656432] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.728934] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7740.995801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7746.022414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7746.214590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7751.670406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.670977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.678125] Lustre: Skipped 1 previous similar message [ 7751.679884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.690967] LustreError: Skipped 1 previous similar message [ 7751.711031] Lustre: Skipped 1 previous similar message [ 7754.967477] Lustre: server umount lustre-OST0000 complete [ 7766.279358] Lustre: server umount lustre-MDT0000 complete [ 7767.017748] LustreError: 171714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7767.048848] LustreError: 171714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7771.008912] LustreError: 171276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846372 with bad export cookie 8725359991088518259 [ 7771.016887] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.031344] LustreError: 171276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7771.412792] Lustre: server umount lustre-MDT0001 complete [ 7781.929908] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:46:21 (1788846381) [ 7790.238515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.179515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7808.139777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.771519] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7820.928221] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7827.140384] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7834.668925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7841.553890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7850.230774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7863.812298] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:47:43 (1788846463) [ 7868.896692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.901192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.908735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7868.908744] Lustre: Skipped 2 previous similar messages [ 7868.938146] Lustre: Skipped 4 previous similar messages [ 7873.729407] Lustre: server umount lustre-OST0000 complete [ 7883.562340] Lustre: server umount lustre-MDT0000 complete [ 7887.490965] LustreError: 174837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846489 with bad export cookie 8725359991088519134 [ 7887.496244] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7887.506043] LustreError: 174837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7887.901844] Lustre: server umount lustre-MDT0001 complete [ 7901.053708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7904.140600] Key type lgssc unregistered [ 7904.492674] LNet: 178586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7904.510026] LNetError: 178586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7904.527446] LNet: Removed LNI 192.168.204.146@tcp [ 7905.750115] Key type .llcrypt unregistered [ 7905.751866] Key type ._llcrypt unregistered [ 7927.144165] Key type ._llcrypt registered [ 7927.145802] Key type .llcrypt registered [ 7927.271652] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7928.644880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7928.861805] alg: No test for adler32 (adler32-zlib) [ 7930.073683] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 7930.540225] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7932.331394] Key type lgssc registered [ 7934.049652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7944.422567] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7944.441645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7960.095512] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7965.215344] LustreError: 179829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 7970.401239] LustreError: 179829:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7970.417800] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7970.425032] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7970.440487] LustreError: 179829:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7970.463405] LustreError: 179829:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7970.477333] LustreError: 179829:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7970.583295] Lustre: server umount lustre-OST0000 complete [ 7970.586883] LustreError: 179829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7979.005120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.503373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7985.147762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7990.765363] LustreError: 180327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.787723] LustreError: 180327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7995.881716] LustreError: 180328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.550962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.436358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.093681] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8016.160265] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8024.034875] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8032.941909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.235428] LustreError: 182147:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8033.248270] Lustre: 182147:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8033.443254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8041.796814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8050.292961] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8054.753636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8054.763660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.780790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.297833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.316406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.337153] Lustre: Skipped 1 previous similar message [ 8059.098579] Lustre: server umount lustre-MDT0000 complete [ 8061.408866] LustreError: 181252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.431192] LustreError: 181252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8062.865704] LustreError: 180308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846664 with bad export cookie 1731658157870749400 [ 8062.887841] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.286187] Lustre: server umount lustre-MDT0001 complete [ 8078.641475] Lustre: server umount lustre-OST0000 complete [ 8090.697154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8104.235554] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8152.669282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.010866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8153.052729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8153.143311] Lustre: lustre-MDT0000: new disk, initializing [ 8153.271945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.300803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8158.375802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8170.961449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.106251] Lustre: 185672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8171.177377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8171.183238] Lustre: Skipped 1 previous similar message [ 8171.258484] Lustre: lustre-MDT0001: new disk, initializing [ 8171.370480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8171.410281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8171.421083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8177.128586] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8188.277978] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8194.801205] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8202.244844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.586701] Lustre: lustre-OST0000: new disk, initializing [ 8202.591453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8202.599898] Lustre: 187640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8202.675829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.525137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8208.553778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8208.680726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8209.565904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8219.994777] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8226.043807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8226.252882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8232.090784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8232.300676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8234.464501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8234.466710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.475725] LustreError: Skipped 1 previous similar message [ 8234.477723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8234.483774] Lustre: Skipped 1 previous similar message [ 8234.501989] Lustre: Skipped 3 previous similar messages [ 8239.588159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.605339] Lustre: Skipped 1 previous similar message [ 8240.823621] Lustre: server umount lustre-OST0000 complete [ 8248.291726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8248.305388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.319397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.918735] Lustre: server umount lustre-MDT0000 complete [ 8254.946274] LustreError: 185683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.962223] LustreError: 185683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8256.643415] LustreError: 185665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846858 with bad export cookie 1731658157870750121 [ 8256.645307] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.664029] LustreError: 185665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8257.129301] Lustre: server umount lustre-MDT0001 complete [ 8267.894546] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:54:27 (1788846867) [ 8290.385638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8294.055979] Key type lgssc unregistered [ 8294.410988] LNet: 190588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8294.421703] LNetError: 190588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8295.463151] LNet: Removed LNI 192.168.204.146@tcp [ 8296.677704] Key type .llcrypt unregistered [ 8296.683526] Key type ._llcrypt unregistered [ 8319.122810] Key type ._llcrypt registered [ 8319.127954] Key type .llcrypt registered [ 8319.273253] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8320.951362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8321.003463] alg: No test for adler32 (adler32-zlib) [ 8322.086371] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8322.442232] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8324.223222] Key type lgssc registered [ 8325.340744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8333.684592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8333.731672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8335.370226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.031282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8345.574569] LustreError: 191803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8345.589719] LustreError: 191803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8350.691878] LustreError: 191802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.596920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.135715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8357.880675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8368.253831] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.685624] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8386.175991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.520208] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8386.522705] LustreError: 193669:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8386.528672] Lustre: 193669:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8386.667588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.979772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8405.363806] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8408.545920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8408.550517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.565843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8411.618394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8411.640414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.656991] Lustre: Skipped 1 previous similar message [ 8414.617358] Lustre: server umount lustre-MDT0000 complete [ 8416.746456] LustreError: 191802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.780147] LustreError: 191802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8418.912587] LustreError: 191782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847020 with bad export cookie 11466697055178036432 [ 8418.934250] LustreError: 191782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8418.935418] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8419.330592] Lustre: server umount lustre-MDT0001 complete [ 8434.522591] Lustre: server umount lustre-OST0000 complete [ 8445.015847] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8457.455484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8503.944224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.202234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8504.243478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8504.382516] Lustre: lustre-MDT0000: new disk, initializing [ 8504.479139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.529359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8509.549859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8522.596845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.757344] Lustre: 197194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8522.834863] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8522.838834] Lustre: Skipped 1 previous similar message [ 8522.849802] LustreError: 197980:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8522.863819] LustreError: 197980:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8522.873706] Lustre: 197980:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8522.941667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8522.949162] Lustre: Skipped 1 previous similar message [ 8523.161606] Lustre: lustre-MDT0001: new disk, initializing [ 8523.359901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8523.415393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8523.431771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8529.179715] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8541.464979] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8549.751276] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8559.393205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.681168] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8559.687083] Lustre: Skipped 1 previous similar message [ 8559.690659] LustreError: 199143:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8559.699956] LustreError: 199143:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8559.709150] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8559.719347] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8559.737711] Lustre: lustre-OST0000: new disk, initializing [ 8559.742428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8559.748919] Lustre: 199166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8559.847963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8567.357024] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8569.481048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8569.496882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8569.621919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8580.211813] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8585.960656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8586.179576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8591.652413] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8591.958236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8595.430086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.440789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8595.461449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8595.470575] LustreError: Skipped 1 previous similar message [ 8595.486638] Lustre: Skipped 3 previous similar messages [ 8600.596988] Lustre: server umount lustre-OST0000 complete [ 8610.273226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8610.281950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8610.291208] Lustre: Skipped 1 previous similar message [ 8610.298501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8612.496235] Lustre: server umount lustre-MDT0000 complete [ 8615.906453] LustreError: 197202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.930962] LustreError: 197202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8616.704776] LustreError: 197188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847218 with bad export cookie 11466697055178037153 [ 8616.713106] LustreError: 197188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8616.720224] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8617.152754] Lustre: server umount lustre-MDT0001 complete [ 8628.451720] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:00:27 (1788847227) [ 8652.936389] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8667.442254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.909791] LustreError: 202328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.956115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.665397] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8677.360057] LustreError: 202329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.378098] LustreError: 202329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8682.284515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.626537] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8690.545241] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8697.883368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8704.662284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8714.489907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8714.727857] Lustre: lustre-OST0001: new disk, initializing [ 8714.731497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8714.738100] Lustre: 204635:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8714.859245] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8714.871402] Lustre: Skipped 2 previous similar messages [ 8723.080512] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8724.635600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8724.650773] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8724.778367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8735.036149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8744.080472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8746.413259] Lustre: DEBUG MARKER: ADD OST3 [ 8752.164679] loop: module loaded [ 8752.257635] loop0: detected capacity change from 0 to 204800000 [ 8752.526473] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8752.552893] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8752.570838] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8759.522492] loop0: detected capacity change from 0 to 204800000 [ 8760.126385] Lustre: DEBUG MARKER: STOP OST3 [ 8766.732352] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.093285] Lustre: lustre-OST0003: new disk, initializing [ 8767.107034] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8767.116435] Lustre: 206657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.700222] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8768.916361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8768.938187] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8768.997701] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8770.722651] LustreError: 206791:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8770.937700] Lustre: Failing over lustre-OST0003 [ 8771.172855] Lustre: server umount lustre-OST0003 complete [ 8773.717571] LustreError: 204857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.727262] LustreError: 204857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8774.120961] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8774.130110] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.151687] Lustre: Skipped 1 previous similar message [ 8775.893101] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8786.069704] Lustre: DEBUG MARKER: START OST3 again [ 8795.145314] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8795.541470] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8795.556982] Lustre: Skipped 1 previous similar message [ 8795.576383] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8796.651354] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8797.132264] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8797.135780] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8802.388492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8825.459292] Lustre: Failing over lustre-OST0003 [ 8825.678512] Lustre: server umount lustre-OST0003 complete [ 8826.349259] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.368889] Lustre: Skipped 2 previous similar messages [ 8826.376226] LustreError: 203815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8826.401542] LustreError: 203815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8846.831948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8846.835435] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.837220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8846.837228] Lustre: Skipped 1 previous similar message [ 8846.881968] Lustre: Skipped 3 previous similar messages [ 8857.063509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.078413] Lustre: Skipped 8 previous similar messages [ 8860.127191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8860.504789] Lustre: server umount lustre-MDT0000 complete [ 8865.140314] LustreError: 202308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847466 with bad export cookie 11466697055178038035 [ 8865.164587] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8871.775299] Lustre: server umount lustre-MDT0001 complete [ 8876.247465] Lustre: server umount lustre-OST0000 complete [ 8880.431115] Lustre: server umount lustre-OST0001 complete [ 8887.841697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8896.799041] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8941.211763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.538796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8941.590445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8941.703175] Lustre: lustre-MDT0000: new disk, initializing [ 8941.867071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.893713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8947.235215] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8962.032454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.298823] Lustre: 212112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8962.452301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8962.459175] Lustre: Skipped 1 previous similar message [ 8962.576153] Lustre: lustre-MDT0001: new disk, initializing [ 8962.799849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8962.825943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8967.865243] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8980.238740] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8987.011482] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8995.186964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.485608] Lustre: lustre-OST0000: new disk, initializing [ 8995.489901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8995.499563] Lustre: 214083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8997.069444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8997.090439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8997.214898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9004.358845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9018.072468] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9024.804923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9025.065990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9031.038449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9031.213540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9038.316703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9038.330854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.333799] LustreError: Skipped 1 previous similar message [ 9038.343052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9038.346988] Lustre: Skipped 3 previous similar messages [ 9038.356983] Lustre: Skipped 7 previous similar messages [ 9039.769404] Lustre: server umount lustre-OST0000 complete [ 9058.272595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9058.512603] Lustre: server umount lustre-MDT0000 complete [ 9058.789350] LustreError: 212124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.814730] LustreError: 212124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9062.790456] LustreError: 212917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847664 with bad export cookie 11466697055178040331 [ 9062.798294] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.803831] LustreError: 212917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9063.339486] Lustre: server umount lustre-MDT0001 complete [ 9067.388121] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:07:47 (1788847667) [ 9083.568732] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9095.961553] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9135.016981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9135.291431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9135.335616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9135.404394] Lustre: lustre-MDT0000: new disk, initializing [ 9135.491903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9140.409403] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9151.660503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9151.779520] Lustre: 218695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9152.066775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9156.899741] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9166.720445] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9173.071272] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9179.449171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9179.710724] Lustre: 220666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9181.328230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9186.518265] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9197.623949] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9202.866850] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9203.039485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9207.466832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9207.680563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9211.876794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.888976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9211.897521] Lustre: Skipped 2 previous similar messages [ 9211.907177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9211.909165] LustreError: Skipped 1 previous similar message [ 9211.940248] Lustre: Skipped 6 previous similar messages [ 9224.159280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9224.366417] Lustre: server umount lustre-OST0000 complete [ 9234.701615] Lustre: server umount lustre-MDT0000 complete [ 9237.473505] LustreError: 218707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.502156] LustreError: 218707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9238.670024] LustreError: 218688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847840 with bad export cookie 11466697055178041213 [ 9238.682179] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.690056] LustreError: 218688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9239.128277] Lustre: server umount lustre-MDT0001 complete [ 9248.216415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.779924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9248.785461] Lustre: Skipped 5 previous similar messages [ 9254.340544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9263.206756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.809542] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9274.361923] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.545962] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9285.947851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.302887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9299.716739] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9312.741717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9312.746092] LustreError: Skipped 1 previous similar message [ 9312.759762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9312.764665] Lustre: Skipped 7 previous similar messages [ 9316.594255] Lustre: server umount lustre-OST0000 complete [ 9335.263335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9337.450100] Lustre: server umount lustre-MDT0000 complete [ 9342.058986] LustreError: 222869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847943 with bad export cookie 11466697055178042088 [ 9342.076511] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.079085] LustreError: 222869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9342.827672] Lustre: server umount lustre-MDT0001 complete [ 9443.094896] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9448.953512] Key type lgssc unregistered [ 9449.365590] LNet: 226547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9449.377538] LNetError: 226547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9450.409318] LNet: Removed LNI 192.168.204.146@tcp [ 9451.951152] Key type .llcrypt unregistered [ 9451.953140] Key type ._llcrypt unregistered [ 9481.508680] Key type ._llcrypt registered [ 9481.511388] Key type .llcrypt registered [ 9481.689704] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9501.714926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9504.065196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9504.160987] alg: No test for adler32 (adler32-zlib) [ 9505.275963] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9505.615126] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9507.409332] Key type lgssc registered [ 9509.059727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9547.530522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9547.549307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.796548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9548.837425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9548.944858] Lustre: lustre-MDT0000: new disk, initializing [ 9549.021077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.056977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9553.728083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9563.862708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.000619] Lustre: 229898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9564.049035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9564.052896] Lustre: Skipped 1 previous similar message [ 9564.111240] Lustre: lustre-MDT0001: new disk, initializing [ 9564.213723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9564.255875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9564.265126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9569.156698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9580.375403] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9587.567546] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9596.973848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9597.290434] Lustre: lustre-OST0000: new disk, initializing [ 9597.298758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9597.310404] Lustre: 231867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9597.396643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9604.916535] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9606.231629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9606.262193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9606.379588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9615.872820] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9622.113687] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9622.499439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9627.755467] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9628.021813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9632.225569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9632.237460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9632.275633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9632.296608] LustreError: Skipped 1 previous similar message [ 9632.323637] Lustre: Skipped 1 previous similar message [ 9636.631369] Lustre: server umount lustre-OST0000 complete [ 9642.469425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.473525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9642.487251] Lustre: Skipped 2 previous similar messages [ 9642.500188] Lustre: Skipped 1 previous similar message [ 9649.504983] Lustre: server umount lustre-MDT0000 complete [ 9652.704133] LustreError: 230992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.736151] LustreError: 230992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9653.939067] LustreError: 229889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848255 with bad export cookie 10500472525618071235 [ 9653.951671] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9653.959403] LustreError: 229889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9654.520629] Lustre: server umount lustre-MDT0001 complete [ 9675.579902] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9687.886710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.501108] LustreError: 234802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.564200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9693.664624] LustreError: 234803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.909065] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9698.798034] LustreError: 234802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.909557] LustreError: 234803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.056933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.385393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9709.504516] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9712.440668] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9719.368395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9719.682933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9726.103409] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9736.144569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.360532] Lustre: lustre-OST0001: new disk, initializing [ 9736.366174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9736.374086] Lustre: 237112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9736.499716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.508788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9746.003136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9746.019381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9746.134994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9756.684367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9766.016060] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9781.394534] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:19:41 (1788848381) [ 9787.366254] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.381284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9787.391747] Lustre: Skipped 2 previous similar messages [ 9787.392240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9787.416917] Lustre: Skipped 3 previous similar messages [ 9790.384247] Lustre: server umount lustre-MDT0000 complete [ 9792.485073] LustreError: 235988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.500040] LustreError: 235988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9795.490125] LustreError: 237880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848397 with bad export cookie 10500472525618072117 [ 9795.510739] LustreError: 237880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9795.514259] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9795.979205] Lustre: server umount lustre-MDT0001 complete [ 9810.825579] Lustre: server umount lustre-OST0000 complete [ 9813.791894] Lustre: 228121:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848399/real 1788848399] req@ffff9659d39ec700 x1875743192760832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788848415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9813.853927] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.876728] Lustre: Skipped 1 previous similar message [ 9814.978892] Lustre: server umount lustre-OST0001 complete [ 9823.045390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9839.786673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9885.418281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.704504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9885.734530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9885.813899] Lustre: lustre-MDT0000: new disk, initializing [ 9885.915853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.951334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9891.727832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9905.427169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.616301] Lustre: 241083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9905.701091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9905.710172] Lustre: Skipped 1 previous similar message [ 9905.857461] Lustre: lustre-MDT0001: new disk, initializing [ 9906.145244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9906.161543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9912.782652] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9924.390574] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9930.804347] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9938.918367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.586679] Lustre: lustre-OST0000: new disk, initializing [ 9939.593715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9939.603785] Lustre: 243054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9941.085960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9941.100536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9941.296609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9950.220183] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9964.430264] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9970.031978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9970.215506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9975.736473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9975.962342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9982.439094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.450142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9982.463538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9984.492751] Lustre: server umount lustre-OST0000 complete [ 9992.674120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9992.677239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9992.698939] Lustre: Skipped 2 previous similar messages [ 9992.706369] Lustre: Skipped 1 previous similar message [ 9994.967221] Lustre: server umount lustre-MDT0000 complete [ 9997.797274] LustreError: 241095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.817573] LustreError: 241095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9998.920018] LustreError: 241074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848600 with bad export cookie 10500472525618074252 [ 9998.927594] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.943038] LustreError: 241074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9999.403039] Lustre: server umount lustre-MDT0001 complete [10008.613978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.335898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10009.340964] Lustre: Skipped 2 previous similar messages [10014.381839] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10023.342116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.603252] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10037.078579] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10042.175102] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10049.542976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10058.433662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10068.314856] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10213.856267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10213.866696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.899850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10213.910938] Lustre: Skipped 1 previous similar message [10218.781912] Lustre: server umount lustre-OST0000 complete [10223.584756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10223.596237] LustreError: Skipped 1 previous similar message [10229.135654] Lustre: server umount lustre-MDT0000 complete [10232.290679] LustreError: 245264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.319815] LustreError: 245264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10233.997752] LustreError: 245252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848835 with bad export cookie 10500472525618075134 [10234.014027] LustreError: 245252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10234.015117] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10234.485978] Lustre: server umount lustre-MDT0001 complete [10335.845331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10338.327379] Key type lgssc unregistered [10338.656987] LNet: 248877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10338.679406] LNetError: 248877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10338.705296] LNet: Removed LNI 192.168.204.146@tcp [10339.564339] Key type .llcrypt unregistered [10339.566957] Key type ._llcrypt unregistered [10365.493961] Key type ._llcrypt registered [10365.495959] Key type .llcrypt registered [10365.579615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10379.648574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10380.798282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10380.849734] alg: No test for adler32 (adler32-zlib) [10381.869915] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10382.079539] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10383.855328] Key type lgssc registered [10384.983152] Lustre: Echo OBD driver; http://www.lustre.org/ [10433.353375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10443.704951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10443.732870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.913335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10444.936151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10444.992706] Lustre: lustre-MDT0000: new disk, initializing [10445.048084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10445.065959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10448.703154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10458.259607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.353417] Lustre: 252942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10458.394231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10458.398084] Lustre: Skipped 1 previous similar message [10458.457365] Lustre: lustre-MDT0001: new disk, initializing [10458.555630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10458.597094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10458.605932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10462.898840] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10467.505982] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10473.734427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.942829] Lustre: lustre-OST0000: new disk, initializing [10473.948277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10473.957108] Lustre: 254576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10474.055828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10476.784095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10476.791866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10476.973457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10481.003414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10491.708954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.912373] Lustre: lustre-OST0001: new disk, initializing [10491.919257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10491.925489] Lustre: 255448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10492.119417] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10499.086628] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10499.155467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10499.162032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10499.292529] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10509.266955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10515.676552] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10525.152398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10525.160202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.163791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10525.181600] Lustre: Skipped 1 previous similar message [10530.274696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10530.279269] Lustre: Skipped 5 previous similar messages [10533.389203] Lustre: server umount lustre-MDT0000 complete [10537.786738] LustreError: 254205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849139 with bad export cookie 16046541097286481090 [10537.805958] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10537.840056] LustreError: 254205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10538.515861] Lustre: server umount lustre-MDT0001 complete [10553.345855] Lustre: server umount lustre-OST0000 complete [10556.256125] Lustre: 250449:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849142/real 1788849142] req@ffff965ac6e30380 x1875744111274880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10556.283184] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.303772] Lustre: Skipped 2 previous similar messages [10556.570817] Lustre: server umount lustre-OST0001 complete [10569.107364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10578.637891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10587.088487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10596.440265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10617.282334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10627.498854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.518975] Lustre: lustre-MDT0000: reset Object Index mappings [10627.881732] LustreError: 260456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.905834] LustreError: 260456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10627.955116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10632.561660] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10633.186238] LustreError: 260457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.285711] LustreError: 260456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.817286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.837054] Lustre: lustre-MDT0001: reset Object Index mappings [10641.236194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10645.301455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10647.837810] Lustre: 261600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10653.999924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10660.677605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10665.573428] LustreError: 261956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.822048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10670.052398] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10670.063272] Lustre: Skipped 1 previous similar message [10673.173987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10673.187760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10678.140733] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10686.611934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10690.202978] Lustre: 263472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10696.676587] Lustre: server umount lustre-MDT0000 complete [10697.696797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10697.704769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10697.719644] LustreError: 261520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.518352] LustreError: 261602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849302 with bad export cookie 16046541097286487873 [10700.542366] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10700.955549] Lustre: server umount lustre-MDT0001 complete [10714.519542] Lustre: server umount lustre-OST0000 complete [10728.994650] Lustre: server umount lustre-OST0001 complete [10747.001558] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:35:47 (1788849347) [10761.989409] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10771.397753] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10802.967896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.229888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10803.260793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10803.316460] Lustre: lustre-MDT0000: new disk, initializing [10803.369521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10803.388565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10807.359466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10816.580188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.690963] Lustre: 267726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10816.736838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10816.740490] Lustre: Skipped 1 previous similar message [10816.793195] Lustre: lustre-MDT0001: new disk, initializing [10816.904471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10816.924461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10820.671375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10829.635571] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10835.517141] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10842.226386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.530218] Lustre: lustre-OST0000: new disk, initializing [10842.534416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10842.540902] Lustre: 269695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.534673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10844.547789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10844.730696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10848.885275] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10857.486364] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10862.810764] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10863.087521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10868.264645] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10868.523443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10875.368142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10875.371927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.377989] LustreError: Skipped 1 previous similar message [10875.391444] Lustre: Skipped 3 previous similar messages [10875.393333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10875.416893] Lustre: Skipped 1 previous similar message [10876.641333] Lustre: server umount lustre-OST0000 complete [10883.566123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10883.580992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.596809] Lustre: Skipped 1 previous similar message [10883.609273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10886.965833] Lustre: server umount lustre-MDT0000 complete [10890.446714] LustreError: 267720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849492 with bad export cookie 16046541097286489966 [10890.452788] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10890.459117] LustreError: 267720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10890.751674] Lustre: server umount lustre-MDT0001 complete [10907.407402] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10915.409597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.708962] LustreError: 272629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.725351] LustreError: 272629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10915.763528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.768772] Lustre: Skipped 2 previous similar messages [10918.917262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10925.402580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.936160] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10931.215264] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10936.302705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10941.562868] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10947.683136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10947.804268] Lustre: lustre-OST0001: new disk, initializing [10947.809980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10947.818657] Lustre: 274935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10949.577108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10949.585094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10949.658262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10952.348600] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10960.642211] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10963.480647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10967.621235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10967.656688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10973.076824] Lustre: server umount lustre-MDT0000 complete [10976.232943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.237500] LustreError: 272624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.244945] Lustre: Skipped 3 previous similar messages [10976.269240] LustreError: 272624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10976.295351] LustreError: 273362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849578 with bad export cookie 16046541097286490855 [10976.307830] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10976.308397] LustreError: 273362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10978.573831] Lustre: server umount lustre-MDT0001 complete [10991.931506] Lustre: server umount lustre-OST0000 complete [11004.457929] Lustre: server umount lustre-OST0001 complete [11018.443484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11026.396248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.655049] LustreError: 277988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.207931] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11036.910472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.776214] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11043.129954] Lustre: 279128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11048.826728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.152731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11049.160476] Lustre: Skipped 5 previous similar messages [11053.994864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11054.287578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11061.060430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11062.341965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11065.896918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11072.226794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11075.187484] Lustre: 281003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11079.642518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11079.729439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11087.672748] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:41:27 (1788849687) [11103.201667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11103.221752] Lustre: Skipped 1 previous similar message [11103.232535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11103.241507] Lustre: Skipped 3 previous similar messages [11108.325099] LustreError: 277984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.357743] LustreError: 277984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11108.404369] Lustre: server umount lustre-MDT0000 complete [11111.720216] LustreError: 277970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849713 with bad export cookie 16046541097286497148 [11111.732228] LustreError: 277970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11111.733293] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.445751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11113.448356] Lustre: Skipped 3 previous similar messages [11126.239150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11126.506612] Lustre: server umount lustre-MDT0001 complete [11144.159160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11144.249050] Lustre: server umount lustre-OST0000 complete [11153.047582] Lustre: server umount lustre-OST0001 complete [11157.868264] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11166.742583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11202.920923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11210.979333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.126563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11211.165744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11211.243831] Lustre: lustre-MDT0000: new disk, initializing [11211.328114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11214.529567] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11222.268761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.354202] Lustre: 285241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11222.553224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11225.919927] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11234.401200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.581291] Lustre: 286874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11235.873751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11239.376075] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11247.496448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.682209] Lustre: 287745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11249.805948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11252.667683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11261.123246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11264.319120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11264.324912] Lustre: Skipped 1 previous similar message [11276.953053] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:44:37 (1788849877) [11278.816683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11278.824696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.841358] Lustre: Skipped 5 previous similar messages [11278.847837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11278.853650] Lustre: Skipped 4 previous similar messages [11284.966840] Lustre: server umount lustre-MDT0000 complete [11285.988559] LustreError: 285253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.003217] LustreError: 285253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11287.758649] LustreError: 285235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849889 with bad export cookie 16046541097286505044 [11287.764979] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11287.773994] LustreError: 285235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11288.083907] Lustre: server umount lustre-MDT0001 complete [11301.890806] Lustre: server umount lustre-OST0000 complete [11315.280708] Lustre: server umount lustre-OST0001 complete [11322.920447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.331407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11323.338066] Lustre: Skipped 5 previous similar messages [11327.105209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11333.835402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.644187] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11342.924513] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11347.218578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11351.936564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.401143] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11362.268066] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11385.475141] loop1: detected capacity change from 0 to 204800000 [11385.656888] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11385.668619] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11385.677609] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11389.495897] loop1: detected capacity change from 0 to 204800000 [11394.146731] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.227667] LustreError: 289978:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16960: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11394.247592] LustreError: 289978:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16960 [11394.256255] LustreError: 289978:0:(mgs_handler.c:517:mgs_target_reg()) fs16960-MDT0001: Error checking target index: rc = -13 [11394.265516] LustreError: fs16960-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11394.271993] LustreError: 293727:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11394.279238] LustreError: 293727:0:(tgt_mount.c:2132:server_put_super()) no obd fs16960-MDT0001 [11394.285332] LustreError: 293727:0:(tgt_mount.c:116:server_deregister_mount()) fs16960-MDT0001 not registered [11394.313382] Lustre: server umount fs16960-MDT0001 complete [11394.316440] LustreError: 293727:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11400.274712] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.342667] Lustre: Setting parameter fs16960-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16960-MDT0001 [11400.380805] Lustre: srv-fs16960-MDT0001: No data found on store. Initialize space. [11400.386527] Lustre: Skipped 4 previous similar messages [11400.441493] Lustre: fs16960-MDT0001: new disk, initializing [11400.443904] Lustre: Skipped 3 previous similar messages [11400.522224] Lustre: srv-fs16960-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11400.526800] Lustre: 294192:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16960-MDT0001-mdtlov: can't create new llog: rc = -115 [11400.538862] LustreError: 294192:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11403.447510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11408.509176] Lustre: server umount fs16960-MDT0001 complete [11417.943977] loop1: detected capacity change from 0 to 204800000 [11422.452218] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11422.705392] Lustre: 295760:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16960-MDT0001-mdtlov: can't create new llog: rc = -115 [11422.715427] LustreError: 295760:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11426.070269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11429.620222] Lustre: server umount fs16960-MDT0001 complete [11436.512394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11436.515706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.523299] Lustre: Skipped 3 previous similar messages [11436.527583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.530697] Lustre: Skipped 4 previous similar messages [11450.335138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11450.474973] Lustre: server umount lustre-MDT0000 complete [11456.986227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.084443] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11460.576351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11462.639861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11466.781330] loop1: detected capacity change from 0 to 204800000 [11471.251928] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.600319] Lustre: 298204:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16960-MDT0001-mdtlov: can't create new llog: rc = -115 [11471.612790] LustreError: 298204:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16960-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11474.714403] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11478.176436] Lustre: server umount fs16960-MDT0001 complete [11493.357087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11493.362845] LustreError: Skipped 1 previous similar message [11503.589351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.603115] Lustre: Skipped 14 previous similar messages [11507.168359] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11507.259739] Lustre: server umount lustre-OST0000 complete [11515.938798] Lustre: server umount lustre-MDT0000 complete [11518.116233] LustreError: 294737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850119 with bad export cookie 16046541097286507725 [11518.119954] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11518.130247] LustreError: 294737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11518.357328] Lustre: server umount lustre-MDT0001 complete [11525.227913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11526.974588] Key type lgssc unregistered [11527.202298] LNet: 300547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11527.206424] LNetError: 300547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11527.221482] LNet: Removed LNI 192.168.204.146@tcp [11527.848167] Key type .llcrypt unregistered [11527.853560] Key type ._llcrypt unregistered [11542.747359] Key type ._llcrypt registered [11542.753503] Key type .llcrypt registered [11542.826894] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11543.508271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11543.517502] alg: No test for adler32 (adler32-zlib) [11544.498287] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11544.639542] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11546.255185] Key type lgssc registered [11547.011316] Lustre: Echo OBD driver; http://www.lustre.org/ [11551.983555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11551.996544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.256896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11556.024208] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11561.677966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.974905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11564.586798] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11569.562504] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11573.622548] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11578.532061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11578.791757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11582.422185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11586.985988] LustreError: 303620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.001580] LustreError: 303620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11587.074671] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11592.162474] LustreError: 303621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11597.285601] LustreError: 303620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.437092] LustreError: 303620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.450437] LustreError: 303620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11608.547422] LustreError: 303621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.554399] LustreError: 303621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11609.626264] loop1: detected capacity change from 0 to 204800000 [11609.710581] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11609.715718] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11609.724891] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11612.982658] loop1: detected capacity change from 0 to 204800000 [11616.489556] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11616.567809] Lustre: fs25669-OST0000: new disk, initializing [11616.571707] Lustre: srv-fs25669-OST0000: No data found on store. Initialize space. [11616.576872] Lustre: 305545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11616.629113] Lustre: fs25669-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11618.790277] LustreError: 303621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.801129] LustreError: 303621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11620.330710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11625.863439] Lustre: server umount fs25669-OST0000 complete [11633.020383] loop1: detected capacity change from 0 to 204800000 [11636.348828] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.492971] Lustre: fs25669-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11639.271749] LustreError: 303622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.279660] LustreError: 303622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11639.935598] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11645.131552] loop2: detected capacity change from 0 to 204800000 [11645.317748] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11645.328626] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11645.334820] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11648.202239] loop2: detected capacity change from 0 to 204800000 [11651.329613] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.386678] LustreError: fs25669-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11651.398855] LustreError: fs25669-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11651.406944] LustreError: 308587:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11651.410547] LustreError: 308587:0:(tgt_mount.c:2132:server_put_super()) no obd fs25669-OST0001 [11651.414381] LustreError: 308587:0:(tgt_mount.c:116:server_deregister_mount()) fs25669-OST0001 not registered [11651.433891] Lustre: server umount fs25669-OST0001 complete [11651.436541] LustreError: 308587:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11657.403718] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.513134] Lustre: fs25669-OST0001: new disk, initializing [11657.518048] Lustre: srv-fs25669-OST0001: No data found on store. Initialize space. [11657.524816] Lustre: 309137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11657.573548] Lustre: fs25669-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11661.807156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11667.697620] Lustre: server umount fs25669-OST0000 complete [11669.990451] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11673.340252] Lustre: server umount fs25669-OST0001 complete [11675.106493] LustreError: 306592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.115773] LustreError: 306592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11680.231554] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11680.307688] loop1: detected capacity change from 0 to 204800000 [11683.422676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11683.536199] Lustre: fs25669-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11686.644334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11690.607552] loop2: detected capacity change from 0 to 204800000 [11693.984529] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11697.829019] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11703.321484] Lustre: server umount fs25669-OST0000 complete [11708.552417] Lustre: server umount fs25669-OST0001 complete [11716.066429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11716.068780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.082937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11716.087068] Lustre: Skipped 1 previous similar message [11721.185613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11721.594290] Lustre: server umount lustre-OST0000 complete [11725.792424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11725.795623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11725.800867] Lustre: Skipped 1 previous similar message [11725.805904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11725.813332] Lustre: Skipped 1 previous similar message [11729.702850] Lustre: server umount lustre-MDT0000 complete [11732.368815] LustreError: 306290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850334 with bad export cookie 1511628509480316195 [11732.372623] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11732.376610] LustreError: 306290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11732.625583] Lustre: server umount lustre-MDT0001 complete [11739.207501] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11740.820717] Key type lgssc unregistered [11741.009830] LNet: 315050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11741.017269] LNetError: 315050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11741.034095] LNet: Removed LNI 192.168.204.146@tcp [11741.556149] Key type .llcrypt unregistered [11741.558875] Key type ._llcrypt unregistered [11761.889882] Key type ._llcrypt registered [11761.892369] Key type .llcrypt registered [11761.962928] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11772.436685] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11773.024462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11773.038725] alg: No test for adler32 (adler32-zlib) [11773.990701] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11774.163893] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11775.800461] Key type lgssc registered [11776.534240] Lustre: Echo OBD driver; http://www.lustre.org/ [11798.168575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11798.193165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.338082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11799.361681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11799.410485] Lustre: lustre-MDT0000: new disk, initializing [11799.464701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11799.475579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11802.088439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11808.889879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.982186] Lustre: 318896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11809.027480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11809.031039] Lustre: Skipped 1 previous similar message [11809.087264] Lustre: lustre-MDT0001: new disk, initializing [11809.181869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11809.216734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11809.223557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11812.265344] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11818.736343] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.451675] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11826.547608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11826.707230] Lustre: lustre-OST0000: new disk, initializing [11826.709922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11826.714245] Lustre: 320863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11826.767553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11830.486951] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11833.384046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11833.395512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11833.486631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11836.642018] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11837.869869] Lustre: Mounted lustre-client - version 2.17.57_117_g424a755 [11838.568449] Lustre: Unmounted lustre-client [11844.065340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11844.065591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11844.069077] LustreError: Skipped 1 previous similar message [11844.069750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11844.080353] Lustre: Skipped 1 previous similar message [11844.089515] Lustre: Skipped 1 previous similar message [11848.164187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.169562] Lustre: Skipped 1 previous similar message [11853.281210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11853.295940] Lustre: Skipped 1 previous similar message [11858.399577] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11858.405836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11858.411703] Lustre: Skipped 1 previous similar message [11858.488926] Lustre: server umount lustre-OST0000 complete [11863.526293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.527544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.542268] Lustre: Skipped 1 previous similar message [11866.922509] Lustre: server umount lustre-MDT0000 complete [11869.256457] LustreError: 320862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850471 with bad export cookie 121924435347218937 [11869.257893] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11869.267323] LustreError: 320862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11869.470619] Lustre: server umount lustre-MDT0001 complete [11876.191742] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11877.889839] Key type lgssc unregistered [11893.472119] LNet: 323025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11893.478898] LNetError: 323025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11970.276875] LNet: Removed LNI 192.168.204.146@tcp [11970.708729] Key type .llcrypt unregistered [11970.710782] Key type ._llcrypt unregistered [11979.439818] Key type ._llcrypt registered [11979.441498] Key type .llcrypt registered [11979.509908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11987.888643] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11988.417371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11988.437085] alg: No test for adler32 (adler32-zlib) [11989.336083] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11989.471970] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11991.095205] Key type lgssc registered [11991.782480] Lustre: Echo OBD driver; http://www.lustre.org/ [12008.626665] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:56:49 (1788850609) [12012.231410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12012.265877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.396877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12013.415291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12013.454765] Lustre: lustre-MDT0000: new disk, initializing [12013.502518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12013.517246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12015.623618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12021.869970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.932996] Lustre: 326215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12021.963485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12021.966596] Lustre: Skipped 1 previous similar message [12022.013578] Lustre: lustre-MDT0001: new disk, initializing [12022.065510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12022.088282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12022.095268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12024.398528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12029.681339] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12032.508476] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12035.785380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12035.936403] Lustre: lustre-OST0000: new disk, initializing [12035.939519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12035.943701] Lustre: 328183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12035.989443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12039.096888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12043.291350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12043.301818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12043.378483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12044.670281] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12053.472340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12053.474428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12053.487976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12053.525051] Lustre: Skipped 1 previous similar message [12058.546370] Lustre: server umount lustre-OST0000 complete [12068.336859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12068.358917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12068.385630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12068.391651] Lustre: Skipped 1 previous similar message [12070.246345] Lustre: server umount lustre-MDT0000 complete [12077.330189] LustreError: 326209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850679 with bad export cookie 13195293784608736896 [12077.336545] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12077.343902] LustreError: 326209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12077.785543] Lustre: server umount lustre-MDT0001 complete [12107.509670] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12113.448052] Key type lgssc unregistered [12114.126694] LNet: 330115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12114.153190] LNetError: 330115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12114.195125] LNet: Removed LNI 192.168.204.146@tcp [12115.900969] Key type .llcrypt unregistered [12115.909472] Key type ._llcrypt unregistered [12143.055584] Key type ._llcrypt registered [12143.058899] Key type .llcrypt registered [12143.182382] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12144.071572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12144.180690] alg: No test for adler32 (adler32-zlib) [12145.238874] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12145.539331] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12147.199388] Key type lgssc registered [12148.300215] Lustre: Echo OBD driver; http://www.lustre.org/ [12155.647193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12155.665973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.084870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12162.054466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12166.627727] LustreError: 331385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12166.638424] LustreError: 331385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12170.526743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12170.986474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12176.096760] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12184.092866] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12190.177992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12197.564121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12198.075811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12203.760075] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12210.756559] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12224.891159] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12226.609767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12233.308409] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [12236.262215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12236.266062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12236.283102] Lustre: Skipped 1 previous similar message [12236.294046] Lustre: Skipped 2 previous similar messages [12238.385953] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [12239.243865] Lustre: server umount lustre-MDT0000 complete [12241.397125] LustreError: 332381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.446391] LustreError: 332381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12243.507051] LustreError: 332535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.275760] LustreError: 331367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850848 with bad export cookie 3919246833309449216 [12246.306156] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12246.307521] LustreError: 331367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12246.498898] LustreError: 331380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.499813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12246.532258] Lustre: Skipped 1 previous similar message [12246.536433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12251.616947] LustreError: 331380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.617123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12251.629350] LustreError: 331380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12251.636850] Lustre: Skipped 1 previous similar message [12252.894892] Lustre: server umount lustre-MDT0001 complete [12257.288534] Lustre: server umount lustre-OST0000 complete [12267.377532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12271.938178] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12274.575294] Lustre: 335575:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [12276.718845] Lustre: server umount MGS complete [12284.996029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.468664] LustreError: 336148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12285.546646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12290.185055] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12298.902820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12299.203876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12304.192628] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12313.578282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12313.956977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12320.791864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12329.446135] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12331.317846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12337.919635] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12339.628468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12344.800472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12344.809708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.827831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12345.834897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12346.967522] Lustre: server umount lustre-OST0000 complete [12350.945858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12350.957560] Lustre: Skipped 1 previous similar message [12353.032232] Lustre: server umount lustre-MDT0000 complete [12355.564469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12355.582171] LustreError: Skipped 1 previous similar message [12355.588134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.609958] LustreError: 336877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.627593] LustreError: 336877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12357.367378] LustreError: 336129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850959 with bad export cookie 3919246833309450056 [12357.387615] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12357.821196] Lustre: server umount lustre-MDT0001 complete [12371.567616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12374.217265] Key type lgssc unregistered [12374.547380] LNet: 339581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12374.562149] LNetError: 339581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12374.615628] LNet: Removed LNI 192.168.204.146@tcp [12375.604438] Key type .llcrypt unregistered [12375.609605] Key type ._llcrypt unregistered [12398.032288] Key type ._llcrypt registered [12398.034366] Key type .llcrypt registered [12398.137970] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12415.646387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12416.670466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12416.699756] alg: No test for adler32 (adler32-zlib) [12417.935782] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12418.200718] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12419.935650] Key type lgssc registered [12421.006116] Lustre: Echo OBD driver; http://www.lustre.org/ [12462.618944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12462.652257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.047678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12464.086835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12464.221861] Lustre: lustre-MDT0000: new disk, initializing [12464.335939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12464.351890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12469.461307] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12481.974944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.218695] Lustre: 342745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12482.346952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12482.352245] Lustre: Skipped 1 previous similar message [12482.485272] Lustre: lustre-MDT0001: new disk, initializing [12482.628193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.686332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12482.709985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12488.140473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12499.813934] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12507.156037] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12515.627492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12515.924291] Lustre: lustre-OST0000: new disk, initializing [12515.930057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12515.943095] Lustre: 344713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12516.059054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12522.259779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12525.153148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12525.168033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12525.296872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12533.816707] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12541.844839] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12542.120875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12549.356318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12549.769995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12556.282031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12556.282870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.300362] LustreError: Skipped 1 previous similar message [12556.325575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12558.921449] Lustre: server umount lustre-OST0000 complete [12564.448221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12564.453722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12564.471940] Lustre: Skipped 1 previous similar message [12564.482970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12564.488891] Lustre: Skipped 1 previous similar message [12566.504717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.512280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12566.524595] Lustre: Skipped 1 previous similar message [12569.552594] Lustre: server umount lustre-MDT0000 complete [12574.040614] LustreError: 342739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851175 with bad export cookie 12839221307095365575 [12574.048782] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.056814] LustreError: 342739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12574.354662] Lustre: server umount lustre-MDT0001 complete [12585.127492] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:06:24 (1788851184) [12604.193716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12614.594814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12657.895682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.081522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12658.102577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12658.151878] Lustre: lustre-MDT0000: new disk, initializing [12658.192738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12658.203067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12663.602774] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12675.462927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12675.601976] Lustre: 349477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12675.680595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12675.688416] Lustre: Skipped 1 previous similar message [12675.804076] Lustre: lustre-MDT0001: new disk, initializing [12675.900519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12675.943647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12675.965034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12681.452645] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12691.996171] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12698.716625] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12706.241670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12706.564353] Lustre: lustre-OST0000: new disk, initializing [12706.567373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12706.571859] Lustre: 351443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12706.635769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12712.493974] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12716.605622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12716.616279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12716.694891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12722.432402] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12763.116629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12763.123642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12763.129360] LustreError: Skipped 1 previous similar message [12763.131155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12763.170850] Lustre: Skipped 1 previous similar message [12766.989793] Lustre: server umount lustre-OST0000 complete [12773.347072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12773.350301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12773.362437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12773.382351] Lustre: Skipped 1 previous similar message [12773.397223] Lustre: Skipped 2 previous similar messages [12778.102817] Lustre: server umount lustre-MDT0000 complete [12778.464530] LustreError: 350638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.492612] LustreError: 350638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12782.257900] LustreError: 349467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851384 with bad export cookie 12839221307095366457 [12782.263428] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12782.273317] LustreError: 349467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12782.543229] Lustre: server umount lustre-MDT0001 complete [12795.385733] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12798.218356] Key type lgssc unregistered [12798.547748] LNet: 353620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12798.561787] LNetError: 353620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12798.587681] LNet: Removed LNI 192.168.204.146@tcp [12799.568322] Key type .llcrypt unregistered [12799.574658] Key type ._llcrypt unregistered [12903.730849] Key type ._llcrypt registered [12903.738060] Key type .llcrypt registered [12903.920149] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12904.300898] Key type .llcrypt unregistered [12904.303775] Key type ._llcrypt unregistered [12988.491958] Key type ._llcrypt registered [12988.496263] Key type .llcrypt registered [12988.577602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12988.798336] Key type .llcrypt unregistered [12988.803617] Key type ._llcrypt unregistered [13020.529654] Key type ._llcrypt registered [13020.535266] Key type .llcrypt registered [13020.686320] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13021.053581] Key type .llcrypt unregistered [13021.055640] Key type ._llcrypt unregistered [13054.706631] Key type ._llcrypt registered [13054.710649] Key type .llcrypt registered [13054.811328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13055.077335] Key type .llcrypt unregistered [13055.080994] Key type ._llcrypt unregistered [13137.666344] Key type ._llcrypt registered [13137.669576] Key type .llcrypt registered [13137.781821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13138.186378] Key type .llcrypt unregistered [13138.188258] Key type ._llcrypt unregistered [13171.652572] Key type ._llcrypt registered [13171.656491] Key type .llcrypt registered [13171.780835] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13172.150099] Key type .llcrypt unregistered [13172.155510] Key type ._llcrypt unregistered [13205.267817] Key type ._llcrypt registered [13205.270438] Key type .llcrypt registered [13205.364791] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13205.753364] Key type .llcrypt unregistered [13205.755128] Key type ._llcrypt unregistered [13238.175517] Key type ._llcrypt registered [13238.179183] Key type .llcrypt registered [13238.304901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13238.648354] Key type .llcrypt unregistered [13238.651328] Key type ._llcrypt unregistered [13275.634295] Key type ._llcrypt registered [13275.635784] Key type .llcrypt registered [13275.793412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13276.148800] Key type .llcrypt unregistered [13276.155596] Key type ._llcrypt unregistered [13308.046889] Key type ._llcrypt registered [13308.052463] Key type .llcrypt registered [13308.182411] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13308.511509] Key type .llcrypt unregistered [13308.514939] Key type ._llcrypt unregistered [13376.045336] Key type ._llcrypt registered [13376.049643] Key type .llcrypt registered [13376.169470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13376.540736] Key type .llcrypt unregistered [13376.554892] Key type ._llcrypt unregistered [13408.922703] Key type ._llcrypt registered [13408.928512] Key type .llcrypt registered [13409.104534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13409.428383] Key type .llcrypt unregistered [13409.432243] Key type ._llcrypt unregistered [13445.687628] Key type ._llcrypt registered [13445.689645] Key type .llcrypt registered [13445.821303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13446.078311] Key type .llcrypt unregistered [13446.081356] Key type ._llcrypt unregistered [13512.958161] Key type ._llcrypt registered [13512.960059] Key type .llcrypt registered [13513.154102] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13528.423762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13529.575192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13529.611377] alg: No test for adler32 (adler32-zlib) [13530.627143] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [13530.824526] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13532.463371] Key type lgssc registered [13533.474759] Lustre: Echo OBD driver; http://www.lustre.org/ [13568.665668] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13568.694851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.954126] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13569.990398] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13570.095710] Lustre: tst250fs-MDT0000: new disk, initializing [13570.163856] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13570.180230] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13574.479534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13585.596160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13585.742724] Lustre: 373905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13585.812738] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13585.822298] Lustre: Skipped 1 previous similar message [13585.876976] Lustre: tst250fs-MDT0001: new disk, initializing [13585.953759] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13585.987875] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13586.000427] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13592.193825] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13598.510871] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13606.144458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13606.575065] Lustre: tst250fs-OST0000: new disk, initializing [13606.582292] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13606.591212] Lustre: 375537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13606.702180] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13607.574829] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13607.586151] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13607.771753] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13613.633296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13625.254934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13625.460988] Lustre: tst250fs-OST0001: new disk, initializing [13625.467428] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13625.476425] Lustre: 376411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13625.626736] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13631.587859] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13631.598202] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13631.732284] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13633.523338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13639.517512] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13642.096849] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13642.100549] Lustre: Skipped 2 previous similar messages [13667.809371] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13667.817155] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13667.826293] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13667.836449] Lustre: Skipped 3 previous similar messages [13667.840302] Lustre: Skipped 2 previous similar messages [13669.229776] Lustre: server umount tst250fs-MDT0000 complete [13672.820267] LustreError: 373899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852274 with bad export cookie 13030021074181730932 [13672.834402] LustreError: 373899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13672.837730] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13672.943329] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13672.948473] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [13672.954416] Lustre: Skipped 1 previous similar message [13672.971604] Lustre: Skipped 2 previous similar messages [13673.374266] Lustre: server umount tst250fs-MDT0001 complete [13680.518851] Lustre: server umount tst250fs-OST0000 complete [13685.240228] Lustre: server umount tst250fs-OST0001 complete [13711.667745] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13724.156348] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13764.633043] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:26:04 (1788852364) [13766.659597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13784.354381] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13795.883435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13830.833462] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13556 sec ========= 03:27:10 (1788852430) [13832.617439] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:27:12 (1788852432) === [13836.331941] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:27:15 (1788852435) === [13884.700492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13887.544271] Key type lgssc unregistered [13887.866720] LNet: 385545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13887.874244] LNetError: 385545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13887.888382] LNet: Removed LNI 192.168.204.146@tcp [13888.719144] Key type .llcrypt unregistered [13888.724513] Key type ._llcrypt unregistered