[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 576499796 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2528MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002092] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007503] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009000] pid_max: default: 32768 minimum: 301 [ 0.009143] LSM: Security Framework initializing [ 0.010046] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.022171] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027465] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029162] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030130] *** VALIDATE tmpfs *** [ 0.032095] *** VALIDATE proc *** [ 0.033252] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035378] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.043340] debug: unmapping init [mem 0xffffffff99c59000-0xffffffff99c60fff] [ 0.046365] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047947] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049037] ... version: 2 [ 0.050025] ... bit width: 48 [ 0.051027] ... generic registers: 4 [ 0.052020] ... value mask: 0000ffffffffffff [ 0.053027] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056618] rcu: Hierarchical SRCU implementation. [ 0.058813] smp: Bringing up secondary CPUs ... [ 0.059605] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.070958] smp: Brought up 1 node, 4 CPUs [ 0.072020] smpboot: Max logical packages: 1 [ 0.073019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130020] node 0 deferred pages initialised in 55ms [ 0.136421] devtmpfs: initialized [ 0.137486] x86/mm: Memory block size: 128MB [ 0.141243] gcov: version magic: 0x41383552 [ 0.144390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146647] pinctrl core: initialized pinctrl subsystem [ 0.147476] [ 0.148009] ************************************************************* [ 0.149018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150019] ** ** [ 0.151012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152014] ** ** [ 0.153013] ** This means that this kernel is built to expose internal ** [ 0.154019] ** IOMMU data structures, which may compromise security on ** [ 0.155012] ** your system. ** [ 0.156013] ** ** [ 0.157016] ** If you see this message and you are not debugging the ** [ 0.158011] ** kernel, report this immediately to your vendor! ** [ 0.159135] ** ** [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ************************************************************* [ 0.162916] NET: Registered protocol family 16 [ 0.164216] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166133] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170064] cpuidle: using governor menu [ 0.172458] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176334] PCI: Using configuration type 1 for base access [ 0.178237] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191122] cryptd: max_cpu_qlen set to 1000 [ 0.192357] ACPI: Added _OSI(Module Device) [ 0.194014] ACPI: Added _OSI(Processor Device) [ 0.196019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206309] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215707] ACPI: Interpreter enabled [ 0.217056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219027] ACPI: Using IOAPIC for interrupt routing [ 0.221131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225914] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242875] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258032] acpiphp: Slot [2] registered [ 0.260135] acpiphp: Slot [5] registered [ 0.262182] acpiphp: Slot [6] registered [ 0.264248] acpiphp: Slot [7] registered [ 0.266202] acpiphp: Slot [8] registered [ 0.267225] acpiphp: Slot [9] registered [ 0.269154] acpiphp: Slot [10] registered [ 0.271220] acpiphp: Slot [3] registered [ 0.272070] acpiphp: Slot [4] registered [ 0.273130] acpiphp: Slot [11] registered [ 0.275139] acpiphp: Slot [12] registered [ 0.277014] acpiphp: Slot [13] registered [ 0.278084] acpiphp: Slot [14] registered [ 0.280088] acpiphp: Slot [15] registered [ 0.281153] acpiphp: Slot [16] registered [ 0.284291] acpiphp: Slot [17] registered [ 0.286185] acpiphp: Slot [18] registered [ 0.288163] acpiphp: Slot [19] registered [ 0.290271] acpiphp: Slot [20] registered [ 0.292231] acpiphp: Slot [21] registered [ 0.294372] acpiphp: Slot [22] registered [ 0.296174] acpiphp: Slot [23] registered [ 0.298346] acpiphp: Slot [24] registered [ 0.300149] acpiphp: Slot [25] registered [ 0.302139] acpiphp: Slot [26] registered [ 0.304137] acpiphp: Slot [27] registered [ 0.306258] acpiphp: Slot [28] registered [ 0.308253] acpiphp: Slot [29] registered [ 0.310185] acpiphp: Slot [30] registered [ 0.312121] acpiphp: Slot [31] registered [ 0.313053] PCI host bridge to bus 0000:00 [ 0.315041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329438] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334214] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338748] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354032] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361114] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369040] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377123] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381255] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385093] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389037] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395075] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425166] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.545023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.566423] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.580025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.594029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.654390] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.664021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.675028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.707017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.719938] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.729013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.777015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.792000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.797018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.802018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.820024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.828711] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.832039] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.834552] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.839195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844054] iommu: Default domain type: Passthrough [ 0.845000] SCSI subsystem initialized [ 0.846194] ACPI: bus type USB registered [ 0.847127] usbcore: registered new interface driver usbfs [ 0.850076] usbcore: registered new interface driver hub [ 0.853100] usbcore: registered new device driver usb [ 0.855180] pps_core: LinuxPPS API ver. 1 registered [ 0.857010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861081] PTP clock support registered [ 0.863412] EDAC MC: Ver: 3.0.0 [ 0.866170] PCI: Using ACPI for IRQ routing [ 0.869137] NetLabel: Initializing [ 0.870009] NetLabel: domain hash size = 128 [ 0.872104] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.874167] NetLabel: unlabeled traffic allowed by default [ 0.877418] vgaarb: loaded [ 0.879070] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.888372] clocksource: Switched to clocksource kvm-clock [ 1.034243] VFS: Disk quotas dquot_6.6.0 [ 1.036377] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.039431] *** VALIDATE ramfs *** [ 1.040899] *** VALIDATE hugetlbfs *** [ 1.042474] pnp: PnP ACPI init [ 1.045336] pnp: PnP ACPI: found 6 devices [ 1.062307] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.066229] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068431] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.070785] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.073270] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.075954] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.079548] NET: Registered protocol family 2 [ 1.081832] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.087632] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.091546] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.097346] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.101383] TCP: Hash tables configured (established 65536 bind 65536) [ 1.104891] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.108620] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.111923] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.115618] NET: Registered protocol family 1 [ 1.118611] RPC: Registered named UNIX socket transport module. [ 1.120969] RPC: Registered udp transport module. [ 1.122604] RPC: Registered tcp transport module. [ 1.124569] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.127369] NET: Registered protocol family 44 [ 1.129264] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.131751] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.134139] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.136872] PCI: CLS 0 bytes, default 64 [ 1.138749] Unpacking initramfs... [ 3.119160] debug: unmapping init [mem 0xffff8abfbcc54000-0xffff8abfbffbffff] [ 3.124728] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.127288] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.129575] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.768049] Initialise system trusted keyrings [ 3.769631] Key type blacklist registered [ 3.771582] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.781928] zbud: loaded [ 3.785653] *** VALIDATE nfs *** [ 3.787168] *** VALIDATE nfs4 *** [ 3.789946] pstore: using deflate compression [ 3.794746] Platform Keyring initialized [ 3.970595] NET: Registered protocol family 38 [ 3.973787] Key type asymmetric registered [ 3.976511] Asymmetric key parser 'x509' registered [ 3.980541] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.986671] io scheduler mq-deadline registered [ 3.989655] io scheduler kyber registered [ 3.991185] io scheduler bfq registered [ 3.993250] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.995976] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.998923] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.001794] ACPI: Power Button [PWRF] [ 4.011426] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.019522] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.045730] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.058765] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.085388] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.119731] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.166291] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.173723] Non-volatile memory driver v1.3 [ 4.174949] Linux agpgart interface v0.103 [ 4.223674] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 4.227621] vda: detected capacity change from 0 to 74731520 [ 4.251320] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.255789] vdb: detected capacity change from 0 to 1073741824 [ 4.274674] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.278186] vdc: detected capacity change from 0 to 2621440000 [ 4.301666] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.305863] vdd: detected capacity change from 0 to 2621440000 [ 4.333822] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.337085] vde: detected capacity change from 0 to 4294967296 [ 4.365155] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.370418] vdf: detected capacity change from 0 to 4294967296 [ 4.380098] libphy: Fixed MDIO Bus: probed [ 4.394538] usbcore: registered new interface driver usbserial_generic [ 4.398267] usbserial: USB Serial support registered for generic [ 4.401314] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.409876] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.412808] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.417567] mousedev: PS/2 mouse device common for all mice [ 4.421651] rtc_cmos 00:05: RTC can wake from S4 [ 4.426504] rtc_cmos 00:05: registered as rtc0 [ 4.428423] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.433668] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.436636] intel_pstate: CPU model not supported [ 4.442253] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.446276] hid: raw HID events driver (C) Jiri Kosina [ 4.450340] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.456758] usbcore: registered new interface driver usbhid [ 4.458937] usbhid: USB HID core driver [ 4.460715] drop_monitor: Initializing network drop monitor service [ 4.463313] Initializing XFRM netlink socket [ 4.465733] NET: Registered protocol family 10 [ 4.469143] Segment Routing with IPv6 [ 4.470869] NET: Registered protocol family 17 [ 4.473265] mpls_gso: MPLS GSO support [ 4.480767] RAS: Correctable Errors collector initialized. [ 4.483082] AVX version of gcm_enc/dec engaged. [ 4.484701] AES CTR mode by8 optimization enabled [ 4.568407] sched_clock: Marking stable (4568332072, 0)->(5734979700, -1166647628) [ 4.572327] registered taskstats version 1 [ 4.574265] Loading compiled-in X.509 certificates [ 4.576248] zswap: loaded using pool lzo/zbud [ 4.607929] Key type big_key registered [ 4.628124] Key type encrypted registered [ 4.629850] ima: No TPM chip found, activating TPM-bypass! [ 4.632195] ima: Allocated hash algorithm: sha1 [ 4.633953] ima: No architecture policies found [ 4.635809] evm: Initialising EVM extended attributes: [ 4.637772] evm: security.selinux [ 4.639183] evm: security.ima [ 4.640318] evm: security.capability [ 4.641720] evm: HMAC attrs: 0x1 [ 4.644360] rtc_cmos 00:05: setting system clock to 2026-09-08 21:28:43 UTC (1788902923) [ 4.652960] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 4.655930] debug: unmapping init [mem 0xffffffff99982000-0xffffffff99c58fff] [ 4.664208] Write protecting the kernel read-only data: 28672k [ 4.670977] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 4.675749] debug: unmapping init [mem 0xffffffff98914000-0xffffffff989fffff] [ 4.724626] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.732284] systemd[1]: Detected virtualization kvm. [ 4.734225] systemd[1]: Detected architecture x86-64. [ 4.736303] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.763914] systemd[1]: No hostname configured. [ 4.765779] systemd[1]: Set hostname to . [ 4.768396] random: systemd: uninitialized urandom read (16 bytes read) [ 4.771392] systemd[1]: Initializing machine ID from random generator. [ 4.893764] random: ln: uninitialized urandom read (6 bytes read) [ 5.068367] random: systemd: uninitialized urandom read (16 bytes read) [ 5.070899] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.076252] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.082353] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.271846] device-mapper: uevent: version 1.0.3 [ 6.274676] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.820151] virtio_net virtio0 ens2: renamed from eth0 [ 7.852696] random: fast init done [ 8.207779] scsi host0: ata_piix [ 8.374845] scsi host1: ata_piix [ 8.383583] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.394652] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.155512] random: crng init done [ 13.158099] random: 7 urandom warning(s) missed due to ratelimiting [ 15.589865] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.861842] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.100835] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.762195] SELinux: Disabled at runtime. [ 23.964413] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.988337] systemd[1]: Detected virtualization kvm. [ 23.992391] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.604537] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.610804] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.623763] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.629473] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.633464] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.668804] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.736336] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.sl[ 26.310270] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.826206] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.888500] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.922485] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.485301] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.567277] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 37.191747] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 39.079296] NFS: Registering the id_resolver key type [ 39.094068] Key type id_resolver registered [ 39.105204] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg427-server login: [ 97.833797] hrtimer: interrupt took 4802614 ns [ 112.031092] libcfs: loading out-of-tree module taints kernel. [ 112.140591] Key type ._llcrypt registered [ 112.142193] Key type .llcrypt registered [ 112.250246] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 132.097739] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 134.028669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 134.043197] alg: No test for adler32 (adler32-zlib) [ 135.682928] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 136.700825] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 138.439166] Key type lgssc registered [ 140.344446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.610580] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.730376] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 214.963839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.045933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.424960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.460918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.656024] Lustre: lustre-MDT0000: new disk, initializing [ 216.785467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.823788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.156714] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.302113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.401131] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.451918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 236.457861] Lustre: Skipped 1 previous similar message [ 236.560258] Lustre: lustre-MDT0001: new disk, initializing [ 236.625070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.657154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.666626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.025723] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.331585] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.616534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.985571] Lustre: lustre-OST0000: new disk, initializing [ 256.993192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 257.007098] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.107865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.445136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.455333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.546427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.544868] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.210188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.481920] Lustre: lustre-OST0001: new disk, initializing [ 284.487877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.499133] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.587395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.668514] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.911567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 293.929187] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 293.981817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.118243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.697042] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 324.008410] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing check_logdir /tmp/testlogs/ [ 330.045970] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing yml_node [ 334.509528] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 337.738450] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 340.449401] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 342.305270] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:34:18 EDT 2026 [ 359.993787] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 361.654321] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 379.872265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 379.888243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.919182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.902769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.907108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.844665] Lustre: server umount lustre-MDT0000 complete [ 388.369615] LustreError: 8461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903307 with bad export cookie 16325483573125215770 [ 388.374012] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.381988] LustreError: 8461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 391.145327] LustreError: 7880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.146697] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.150254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.150263] Lustre: Skipped 3 previous similar messages [ 391.160759] LustreError: 7880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 391.217139] Lustre: Skipped 3 previous similar messages [ 393.187568] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.190201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.203821] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 393.215199] Lustre: Skipped 2 previous similar messages [ 394.731531] Lustre: server umount lustre-MDT0001 complete [ 399.371512] Lustre: server umount lustre-OST0000 complete [ 404.367375] Lustre: server umount lustre-OST0001 complete [ 413.327652] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 427.151387] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 468.879938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.245711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 469.296666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 469.374167] Lustre: lustre-MDT0000: new disk, initializing [ 469.505311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.533895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 475.894951] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 488.470614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.601842] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 488.665714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 488.674067] Lustre: Skipped 1 previous similar message [ 488.723986] Lustre: lustre-MDT0001: new disk, initializing [ 488.805237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 488.928075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 488.953595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 494.114081] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 503.254688] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.474269] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.844744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 514.069192] Lustre: lustre-OST0000: new disk, initializing [ 514.072519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 514.080595] Lustre: 17069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 514.162479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.806363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 515.816678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 515.924690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 519.753322] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 527.211699] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.732490] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 530.882777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 534.063692] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 534.256638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 536.549406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.551691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.567068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.572324] LustreError: Skipped 1 previous similar message [ 536.578150] Lustre: Skipped 1 previous similar message [ 546.791165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.795426] Lustre: Skipped 3 previous similar messages [ 550.367273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.458329] Lustre: server umount lustre-OST0000 complete [ 555.488547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.495352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.501402] Lustre: Skipped 1 previous similar message [ 567.263643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.265384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.276218] Lustre: Skipped 6 previous similar messages [ 567.371089] Lustre: server umount lustre-MDT0000 complete [ 569.390317] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903488 with bad export cookie 16325483573125217310 [ 569.391468] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.397694] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 569.575351] Lustre: server umount lustre-MDT0001 complete [ 573.020310] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:38:10 (1788903490) [ 583.082374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.316883] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.360355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.004867] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 588.770104] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.357962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.122352] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 597.694973] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.083811] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.379278] Lustre: Failing over lustre-MDT0000 [ 604.532640] Lustre: server umount lustre-MDT0000 complete [ 606.178362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.179680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.183189] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.188545] Lustre: Skipped 2 previous similar messages [ 615.393819] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.403991] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 619.086122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.174433] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.353863] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 621.320486] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 624.611384] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 624.617075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 624.633947] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 626.464384] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 629.729560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.730069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.738510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.741907] Lustre: Skipped 1 previous similar message [ 643.040012] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 643.115501] Lustre: server umount lustre-MDT0000 complete [ 644.896053] LustreError: 20679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903563 with bad export cookie 16325483573125218724 [ 644.899188] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.901249] LustreError: 20679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 645.062280] Lustre: server umount lustre-MDT0001 complete [ 649.227143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.458299] LustreError: 23275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.505697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.508671] Lustre: Skipped 2 previous similar messages [ 651.614490] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 656.162267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.385386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 661.947782] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.411111] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.491230] Lustre: Failing over lustre-MDT0000 [ 665.498605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.504891] Lustre: Skipped 1 previous similar message [ 665.620286] Lustre: server umount lustre-MDT0000 complete [ 679.936556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.006197] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.148393] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 682.043357] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 682.146677] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 682.149950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 682.153308] Lustre: Skipped 1 previous similar message [ 682.163156] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 686.983302] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 692.704236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.767700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.772742] Lustre: Skipped 11 previous similar messages [ 695.599539] Lustre: server umount lustre-MDT0000 complete [ 697.297647] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903616 with bad export cookie 16325483573125219662 [ 697.299237] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.305645] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 697.469566] Lustre: server umount lustre-MDT0001 complete [ 701.407637] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:40:18 (1788903618) [ 709.127539] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 713.331592] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 731.655519] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 737.383689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.505269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 737.534555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 737.580404] Lustre: lustre-MDT0000: new disk, initializing [ 737.626401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 739.622521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 745.598341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.654705] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 745.822872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 747.736175] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 752.598816] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 755.016621] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 757.997950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.124888] Lustre: 31604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 759.452674] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 759.482322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 759.490827] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 760.867064] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 765.209801] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 769.038487] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 780.257061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 780.257433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.259851] LustreError: Skipped 1 previous similar message [ 780.264447] Lustre: Skipped 2 previous similar messages [ 782.589450] Lustre: server umount lustre-OST0000 complete [ 790.370073] Lustre: server umount lustre-MDT0000 complete [ 790.496726] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.507390] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 791.906641] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903710 with bad export cookie 16325483573125220068 [ 791.910720] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.911686] LustreError: 29632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 792.097774] Lustre: server umount lustre-MDT0001 complete [ 796.439288] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 797.643213] Key type lgssc unregistered [ 797.798835] LNet: 33560:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.802401] LNetError: 33560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.812590] LNet: Removed LNI 192.168.204.127@tcp [ 798.237236] Key type .llcrypt unregistered [ 798.238838] Key type ._llcrypt unregistered [ 808.632605] Key type ._llcrypt registered [ 808.633952] Key type .llcrypt registered [ 808.680966] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 809.140329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.152231] alg: No test for adler32 (adler32-zlib) [ 810.039699] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 810.162563] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 811.759468] Key type lgssc registered [ 812.227532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 815.442972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 815.455169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.665225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.337255] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 821.932387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.083386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.788469] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 827.082791] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 829.325966] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 832.025144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 832.152622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.358581] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 837.127787] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 846.507346] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 852.881132] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:42:50 (1788903770) [ 853.677581] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 854.426601] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 856.912583] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:42:54 (1788903774) [ 860.213413] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:42:57 (1788903777) [ 860.576919] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 863.594326] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 863.596819] Lustre: Skipped 2 previous similar messages [ 864.241147] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:43:02 (1788903782) [ 868.178107] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 868.180934] Lustre: Skipped 7 previous similar messages [ 874.034539] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:43:11 (1788903791) [ 875.187327] Lustre: *** cfs_fail_loc=131b, val=2*** [ 876.632690] Lustre: *** cfs_fail_loc=131b, val=2*** [ 880.060929] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:43:17 (1788903797) [ 880.915899] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 880.918593] Lustre: Skipped 6 previous similar messages [ 885.920219] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:43:23 (1788903803) [ 892.897052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 892.899476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.900634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.904238] Lustre: Skipped 1 previous similar message [ 892.908934] Lustre: Skipped 1 previous similar message [ 896.340945] Lustre: server umount lustre-OST0000 complete [ 898.017137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.018321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.022542] Lustre: Skipped 1 previous similar message [ 903.136226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.139360] Lustre: Skipped 2 previous similar messages [ 908.256135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.258539] Lustre: Skipped 1 previous similar message [ 912.352262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.352279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 912.355350] Lustre: Skipped 1 previous similar message [ 912.425414] Lustre: server umount lustre-MDT0000 complete [ 913.671229] LustreError: 37888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903832 with bad export cookie 6171564133166207835 [ 913.675747] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.676471] LustreError: 37888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 913.783668] Lustre: server umount lustre-MDT0001 complete [ 917.301854] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 918.318025] Key type lgssc unregistered [ 918.450436] LNet: 42807:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.452708] LNetError: 42807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.464514] LNet: Removed LNI 192.168.204.127@tcp [ 918.768282] Key type .llcrypt unregistered [ 918.769629] Key type ._llcrypt unregistered [ 924.626812] Key type ._llcrypt registered [ 924.628652] Key type .llcrypt registered [ 924.662861] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 930.714577] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 931.067909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.095386] alg: No test for adler32 (adler32-zlib) [ 931.941754] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 932.024344] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 933.599152] Key type lgssc registered [ 933.951566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.112589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.119514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.197297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 946.208196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 946.242058] Lustre: lustre-MDT0000: new disk, initializing [ 946.260287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.265723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 947.546767] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 951.536155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.570224] Lustre: 45929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 951.589425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 951.590722] Lustre: Skipped 1 previous similar message [ 951.623756] Lustre: lustre-MDT0001: new disk, initializing [ 951.652266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 951.663156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 951.666549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 952.908058] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 956.502705] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.362405] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.509071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.603824] Lustre: lustre-OST0000: new disk, initializing [ 960.605340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 960.607575] Lustre: 47898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 960.633317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.440643] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 964.663285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 964.666304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 964.697249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 966.189830] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 967.546087] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 967.610697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 968.948110] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 969.013590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 969.696528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.696860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.697996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.699833] LustreError: Skipped 1 previous similar message [ 969.704776] Lustre: Skipped 1 previous similar message [ 971.728510] Lustre: server umount lustre-OST0000 complete [ 974.816779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.817439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.821349] Lustre: Skipped 1 previous similar message [ 974.823987] Lustre: Skipped 1 previous similar message [ 979.088459] Lustre: server umount lustre-MDT0000 complete [ 980.316985] LustreError: 49437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903899 with bad export cookie 1197448360037950712 [ 980.320238] LustreError: 49437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 980.320808] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.424240] Lustre: server umount lustre-MDT0001 complete [ 983.460362] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:45:01 (1788903901) [ 985.526521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.666125] LustreError: 50355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.671581] LustreError: 50355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 985.684503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.022910] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 989.843152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.966397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.246838] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 993.619105] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 995.380241] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.595875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.643521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1002.126302] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1014.560065] LustreError: 52907:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1014.871130] Lustre: Modifying parameter general.timeout=1 in log params [ 1016.010697] Lustre: Modifying parameter general.timeout=5 in log params [ 1016.012665] Lustre: Skipped 3 previous similar messages [ 1018.264055] Lustre: Modifying parameter general.timeout=13 in log params [ 1018.266504] Lustre: Skipped 7 previous similar messages [ 1022.480086] Lustre: Modifying parameter general.timeout=28 in log params [ 1022.481917] Lustre: Skipped 14 previous similar messages [ 1030.711990] Lustre: Modifying parameter general.timeout=56 in log params [ 1030.713774] Lustre: Skipped 27 previous similar messages [ 1046.842867] Lustre: Modifying parameter general.timeout=114 in log params [ 1046.845106] Lustre: Skipped 57 previous similar messages [ 1054.520141] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:46:12 (1788903972) [ 1055.371404] Lustre: 59797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1056.516862] Lustre: 60006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1056.522849] Lustre: 60006:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1066.412947] Lustre: 61636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1066.416817] Lustre: 61636:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 3 previous similar messages [ 1066.418769] LustreError: 61636:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1067.624486] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:46:25 (1788903985) [ 1072.095761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.097785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.101457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.680256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.681484] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.202822] Lustre: server umount lustre-MDT0000 complete [ 1078.341371] LustreError: 52957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903997 with bad export cookie 1197448360037951580 [ 1078.345447] LustreError: 52957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.345885] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1078.450654] Lustre: server umount lustre-MDT0001 complete [ 1089.945318] Lustre: server umount lustre-OST0000 complete [ 1098.093357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.179812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1098.190217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1098.192881] Lustre: Skipped 38 previous similar messages [ 1098.274566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.276967] Lustre: Skipped 1 previous similar message [ 1099.537095] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1102.184528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.214333] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1102.223833] Lustre: 64159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1103.605186] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1105.794830] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.472641] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.526180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.590010] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1111.368794] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1113.808381] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.573881] Lustre: 66724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1124.832412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.834281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.835859] LustreError: Skipped 1 previous similar message [ 1124.838661] Lustre: Skipped 1 previous similar message [ 1124.839282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.842847] Lustre: Skipped 3 previous similar messages [ 1127.562672] Lustre: server umount lustre-OST0000 complete [ 1135.072165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.074397] Lustre: Skipped 3 previous similar messages [ 1143.263130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1143.319371] Lustre: server umount lustre-MDT0000 complete [ 1144.288714] LustreError: 64171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.295117] LustreError: 64171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1144.566141] LustreError: 64900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904063 with bad export cookie 1197448360037953834 [ 1144.568493] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.570402] LustreError: 64900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1144.686879] Lustre: server umount lustre-MDT0001 complete [ 1148.198541] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1149.186284] Key type lgssc unregistered [ 1149.312384] LNet: 67930:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.315236] LNetError: 67930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.325319] LNet: Removed LNI 192.168.204.127@tcp [ 1149.608367] Key type .llcrypt unregistered [ 1149.609445] Key type ._llcrypt unregistered [ 1158.550476] Key type ._llcrypt registered [ 1158.551926] Key type .llcrypt registered [ 1158.585439] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1158.927791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.967383] alg: No test for adler32 (adler32-zlib) [ 1159.811141] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1159.903820] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1161.479140] Key type lgssc registered [ 1161.801613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.327870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.335050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.490582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.797022] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1169.420126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.597705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.868747] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1173.023592] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1174.742659] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1176.921750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.098804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.873424] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1181.235168] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1194.553405] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1200.610307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.611710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.615516] Lustre: Skipped 2 previous similar messages [ 1200.617428] Lustre: Skipped 1 previous similar message [ 1200.621230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1202.957815] Lustre: server umount lustre-MDT0000 complete [ 1204.098275] LustreError: 69300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904122 with bad export cookie 3794949324027162810 [ 1204.102463] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.203832] Lustre: server umount lustre-MDT0001 complete [ 1215.646567] Lustre: server umount lustre-OST0000 complete [ 1224.419508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.518333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1224.527713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1224.609675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.905142] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1228.606881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.637469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1228.649352] Lustre: 74204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1228.820512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.160744] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1232.408141] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1234.147623] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.629338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.700166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1236.845803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.046792] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1242.241701] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1243.088287] Lustre: 76776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1244.048227] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1250.785031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.795127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.805296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.817319] Lustre: Skipped 1 previous similar message [ 1250.834406] Lustre: Skipped 2 previous similar messages [ 1253.361681] Lustre: server umount lustre-OST0000 complete [ 1259.489024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1259.494632] LustreError: Skipped 1 previous similar message [ 1259.497832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.507415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.392657] Lustre: server umount lustre-MDT0000 complete [ 1265.373199] LustreError: 74195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904184 with bad export cookie 3794949324027163622 [ 1265.382107] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.386908] LustreError: 74195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1265.594807] Lustre: server umount lustre-MDT0001 complete [ 1272.401572] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1273.950168] Key type lgssc unregistered [ 1274.105721] LNet: 77981:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.109919] LNetError: 77981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.121148] LNet: Removed LNI 192.168.204.127@tcp [ 1274.511190] Key type .llcrypt unregistered [ 1274.512522] Key type ._llcrypt unregistered [ 1291.637622] Key type ._llcrypt registered [ 1291.639195] Key type .llcrypt registered [ 1291.685291] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1292.143034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.150777] alg: No test for adler32 (adler32-zlib) [ 1293.022731] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1293.158214] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1294.751172] Key type lgssc registered [ 1295.218693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.730228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.738176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.960725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.380470] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1309.173287] LustreError: 79373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.231033] LustreError: 79373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1313.181801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.000112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.765573] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1330.156407] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1337.749098] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1347.988507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.680188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.269557] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1380.111787] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.625265] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1388.717558] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1389.816538] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1392.239774] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1392.249628] Lustre: Skipped 1 previous similar message [ 1396.349884] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1396.356261] Lustre: Skipped 3 previous similar messages [ 1404.525962] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1404.531882] Lustre: Skipped 7 previous similar messages [ 1420.583157] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1420.589861] Lustre: Skipped 17 previous similar messages [ 1453.060875] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1453.063578] Lustre: Skipped 30 previous similar messages [ 1493.577437] Lustre: 87482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1496.531866] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:53:33 (1788904413) [ 1498.318915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1500.230059] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:53:36 (1788904416) [ 1507.299144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.311535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.328066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.810912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.822540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.849489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.692633] Lustre: server umount lustre-OST0000 complete [ 1518.053784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.059132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.073835] Lustre: Skipped 1 previous similar message [ 1518.084895] Lustre: Skipped 1 previous similar message [ 1523.168939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.180991] Lustre: Skipped 1 previous similar message [ 1527.264778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.335276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.501593] Lustre: server umount lustre-MDT0000 complete [ 1532.387348] LustreError: 79392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.399236] LustreError: 80114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904453 with bad export cookie 6242278258489885172 [ 1534.415548] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.415952] LustreError: 80114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1534.733616] Lustre: server umount lustre-MDT0001 complete [ 1545.546358] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1548.578533] Key type lgssc unregistered [ 1548.950798] LNet: 88732:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.956800] LNetError: 88732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.980532] LNet: Removed LNI 192.168.204.127@tcp [ 1549.971649] Key type .llcrypt unregistered [ 1549.978192] Key type ._llcrypt unregistered [ 1554.730231] Key type ._llcrypt registered [ 1554.732018] Key type .llcrypt registered [ 1554.829211] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1560.285128] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules [ 1561.436506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.555703] alg: No test for adler32 (adler32-zlib) [ 1562.739530] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1563.016292] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1564.711279] Key type lgssc registered [ 1566.691784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.177712] LustreError: 89984:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1590.046504] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1592.010935] LustreError: 89984:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1600.109208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1600.124077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.624981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.609874] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1611.745076] LustreError: 91014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.766306] LustreError: 91014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1614.865714] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:55:30 (1788904530) [ 1616.876272] LustreError: 91015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.062437] LustreError: 91046:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1621.073258] LustreError: 91046:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1621.090989] LustreError: 91046:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 1621.579302] Lustre: server umount lustre-MDT0000 complete [ 1637.567431] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1640.844957] Key type lgssc unregistered [ 1641.124858] LNet: 92540:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.138391] LNetError: 92540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.176850] LNet: Removed LNI 192.168.204.127@tcp [ 1641.846722] Key type .llcrypt unregistered [ 1641.849366] Key type ._llcrypt unregistered [ 1660.624485] Key type ._llcrypt registered [ 1660.628063] Key type .llcrypt registered [ 1660.827422] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1662.465501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1662.499564] alg: No test for adler32 (adler32-zlib) [ 1663.664419] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1663.932917] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1665.591178] Key type lgssc registered [ 1667.109676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.412242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.443063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.320163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.001453] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1687.523259] LustreError: 93761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.537814] LustreError: 93761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1692.653057] LustreError: 93762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.102586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.057554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.833336] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1711.814653] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.348385] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.488906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.317273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.729475] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1741.207250] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.062470] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:58:47 (1788904727) [ 1812.910416] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1815.019337] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:58:51 (1788904731) [ 1819.617343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1819.618838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.623947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.650140] Lustre: Skipped 1 previous similar message [ 1824.742462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1824.747818] Lustre: Skipped 4 previous similar messages [ 1825.022766] Lustre: server umount lustre-MDT0000 complete [ 1827.723707] LustreError: 93740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904746 with bad export cookie 540944006648309886 [ 1827.724875] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.732990] LustreError: 93740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1827.991899] Lustre: server umount lustre-MDT0001 complete [ 1841.062629] Lustre: server umount lustre-OST0000 complete [ 1848.810683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.220625] LustreError: 98080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.392843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.083682] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1854.434225] LustreError: 98081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.015226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.500930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.136123] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1865.724711] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.665206] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1875.782500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.921088] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1875.929600] LustreError: 98067:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1875.955981] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1875.963261] LustreError: 100161:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 1875.969590] LustreError: 100161:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 1875.976382] LustreError: 100161:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1876.030300] Lustre: server umount lustre-OST0000 complete [ 1876.033317] LustreError: 100161:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1880.786544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.894537] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1880.898336] LustreError: 98067:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1880.903833] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1880.909304] LustreError: 100621:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 1880.913507] LustreError: 100621:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 1880.917392] LustreError: 100621:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1880.961253] Lustre: server umount lustre-OST0000 complete [ 1880.963194] LustreError: 100621:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1886.144645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.277192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1886.281745] Lustre: Found index 0 for lustre-OST0000, updating log [ 1886.290481] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1886.340094] Lustre: lustre-OST0000: new disk, initializing [ 1886.343727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1886.347816] Lustre: 101158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1886.494196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.898276] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1892.272957] Lustre: Failing over lustre-OST0000 [ 1892.395838] Lustre: server umount lustre-OST0000 complete [ 1899.489298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.490047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1899.490532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1899.495043] Lustre: Skipped 2 previous similar messages [ 1900.787125] Lustre: server umount lustre-MDT0000 complete [ 1902.790170] LustreError: 101156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904821 with bad export cookie 540944006648312623 [ 1902.790170] LustreError: 98062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904821 with bad export cookie 540944006648312623 [ 1902.792833] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.796853] LustreError: 101156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1902.986541] Lustre: server umount lustre-MDT0001 complete [ 1907.311950] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:00:24 (1788904824) [ 1915.916750] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1921.516610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.751299] LustreError: 103500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.803821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.797706] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1928.482388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.924761] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1932.652064] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1936.497546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.914984] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1944.683590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.730027] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1944.789216] Lustre: lustre-OST0001: new disk, initializing [ 1944.791976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1944.796612] Lustre: 105829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1945.077726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1945.082684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1945.121660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1948.009366] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1952.415253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1957.858146] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1962.466495] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.468925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1962.471403] Lustre: Skipped 2 previous similar messages [ 1962.473960] Lustre: Skipped 2 previous similar messages [ 1967.586019] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1967.589453] Lustre: Skipped 4 previous similar messages [ 1974.239137] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1974.320703] Lustre: server umount lustre-MDT0001 complete [ 1974.784821] LustreError: 103495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.794305] LustreError: 103495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1979.267782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.344870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.347783] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1979.377514] Lustre: 103488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.775821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.780916] Lustre: Skipped 3 previous similar messages [ 1981.097562] LustreError: 107485:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1981.101648] LustreError: 107485:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1981.106437] LustreError: 103492:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1981.154871] LustreError: 107500:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -17 [ 1981.159142] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1981.159142] [ 1981.767888] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1984.999393] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1985.732544] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:01:43 (1788904903) [ 1993.043888] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2002.070066] Lustre: 109438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2013.532791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2015.111360] Lustre: 110571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2072.032672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2072.033541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2072.034095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2072.039297] Lustre: Skipped 3 previous similar messages [ 2072.043254] Lustre: Skipped 6 previous similar messages [ 2073.329181] Lustre: server umount lustre-MDT0000 complete [ 2074.721647] LustreError: 105828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904993 with bad export cookie 540944006648313547 [ 2074.722742] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.726791] LustreError: 105828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2074.842605] Lustre: server umount lustre-MDT0001 complete [ 2086.413708] Lustre: server umount lustre-OST0000 complete [ 2098.140814] Lustre: server umount lustre-OST0001 complete [ 2100.084559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2103.353621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2106.455189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2109.824801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2116.540759] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2120.631342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.643215] Lustre: lustre-MDT0000: reset Object Index mappings [ 2120.816620] LustreError: 114514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.824513] LustreError: 114514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2120.840916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.391971] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2125.558275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.572535] Lustre: lustre-MDT0001: reset Object Index mappings [ 2127.462937] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2128.658048] Lustre: 115661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2131.403548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2132.714596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2134.027995] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2135.789465] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2137.411391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.666248] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2142.697516] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2142.698604] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2143.011338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2144.298624] Lustre: 117536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2161.045742] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:04:38 (1788905078) [ 2192.352734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.355624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.360310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.362256] Lustre: Skipped 1 previous similar message [ 2196.321738] Lustre: server umount lustre-MDT0000 complete [ 2197.582204] LustreError: 114497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905116 with bad export cookie 540944006648360328 [ 2197.585935] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.586207] LustreError: 114497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2199.008869] LustreError: 115938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.014848] LustreError: 115938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2203.843630] Lustre: server umount lustre-MDT0001 complete [ 2211.268822] Lustre: server umount lustre-OST0000 complete [ 2218.789146] Lustre: server umount lustre-OST0001 complete [ 2221.023832] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2224.481403] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2237.268279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.347773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2237.357932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2237.399913] Lustre: lustre-MDT0000: new disk, initializing [ 2237.423840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2238.830374] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2241.875826] Lustre: server umount lustre-MDT0000 complete [ 2245.125838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.233968] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2245.239681] Lustre: 121791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2245.244587] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2246.804653] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2248.656540] Lustre: server umount lustre-MDT0000 complete [ 2251.745922] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:06:09 (1788905169) [ 2252.311252] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2252.928438] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:06:10 (1788905170) [ 2258.724415] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2262.213457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.361722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.364035] Lustre: Skipped 5 previous similar messages [ 2263.765317] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2266.673305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.718989] Lustre: 123840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2266.741621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2266.743240] Lustre: Skipped 1 previous similar message [ 2266.778753] Lustre: lustre-MDT0001: new disk, initializing [ 2266.827120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2266.831237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2268.214517] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2270.528366] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2272.783275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2272.888876] Lustre: lustre-OST0000: new disk, initializing [ 2272.892060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2272.896333] Lustre: 125417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2274.172826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2274.177102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2274.209769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2274.907771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2279.094222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.158989] Lustre: 126289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2280.643777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2280.648045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2280.686429] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2281.265578] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2285.850219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2287.105280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2421.974390] Lustre: 132661:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2421.978491] Lustre: 132661:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2421.980892] Lustre: 132661:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2421.983331] Lustre: 132661:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2421.985506] Lustre: 132661:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2421.989208] Lustre: 132661:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2511.922860] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:10:29 (1788905429) [ 2526.692860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.695618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.697863] Lustre: Skipped 8 previous similar messages [ 2526.702588] Lustre: Skipped 11 previous similar messages [ 2529.599527] Lustre: server umount lustre-MDT0000 complete [ 2530.903287] LustreError: 123873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905449 with bad export cookie 540944006648415334 [ 2530.906250] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.907592] LustreError: 123873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2531.808319] LustreError: 123846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.814612] LustreError: 123846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2537.114821] Lustre: server umount lustre-MDT0001 complete [ 2544.639970] Lustre: server umount lustre-OST0000 complete [ 2551.897469] Lustre: server umount lustre-OST0001 complete [ 2553.973724] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2557.078294] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2569.321968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.407413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2569.419563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2569.421505] Lustre: Skipped 1 previous similar message [ 2569.458209] Lustre: lustre-MDT0000: new disk, initializing [ 2569.459416] Lustre: Skipped 1 previous similar message [ 2569.480272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.482619] Lustre: Skipped 3 previous similar messages [ 2569.487964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2570.873756] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2574.985386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.023293] Lustre: 135675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2575.122432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2576.437260] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2580.104973] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2581.938606] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2584.209625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.306950] Lustre: 137644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2585.848354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2585.851477] Lustre: Skipped 1 previous similar message [ 2585.854608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2585.885549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2586.243607] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2589.939537] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2592.981471] Lustre: lustre-MDD0000: changelog on [ 2593.894867] Lustre: lustre-MDD0001: changelog on [ 4329.032459] Lustre: lustre-MDD0001: changelog off [ 4330.372073] Lustre: 135683:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4330.965955] Lustre: lustre-MDD0000: changelog off [ 4335.075093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.076435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.084964] Lustre: Skipped 3 previous similar messages [ 4335.087866] Lustre: Skipped 4 previous similar messages [ 4341.036872] Lustre: server umount lustre-OST0000 complete [ 4345.318133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.332997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.337256] Lustre: Skipped 4 previous similar messages [ 4350.191390] Lustre: server umount lustre-MDT0000 complete [ 4350.440647] LustreError: 135683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.471864] LustreError: 135683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4353.643112] LustreError: 135667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907272 with bad export cookie 540944006649217597 [ 4353.645245] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.649947] LustreError: 135667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4353.937663] Lustre: server umount lustre-MDT0001 complete [ 4372.581092] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4376.440685] Key type lgssc unregistered [ 4376.910698] LNet: 146737:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.921883] LNetError: 146737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.956736] LNet: Removed LNI 192.168.204.127@tcp [ 4378.162078] Key type .llcrypt unregistered [ 4378.172237] Key type ._llcrypt unregistered [ 4406.333734] Key type ._llcrypt registered [ 4406.337980] Key type .llcrypt registered [ 4406.546244] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4425.451801] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4427.006329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4427.145179] alg: No test for adler32 (adler32-zlib) [ 4428.214277] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4428.500546] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4430.215172] Key type lgssc registered [ 4431.463225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4467.791924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4467.808743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.050869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4469.077068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4469.175526] Lustre: lustre-MDT0000: new disk, initializing [ 4469.230642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.247108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4473.564845] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4483.724352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.817184] Lustre: 149939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4483.868905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4483.872845] Lustre: Skipped 1 previous similar message [ 4483.951238] Lustre: lustre-MDT0001: new disk, initializing [ 4484.057257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.100740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4484.119974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4488.365065] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4496.911468] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4502.387250] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.483564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.904875] Lustre: lustre-OST0000: new disk, initializing [ 4509.908365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4509.914782] Lustre: 151908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4510.078743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.615800] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4518.459628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4518.473400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4518.613239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4526.234754] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4532.364237] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_module obdecho/obdecho [ 4534.692224] LustreError: 152910:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4534.707127] LustreError: 152910:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4536.889849] LustreError: 153006:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4550.624622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4550.644661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.670345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.672521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.676764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.686659] Lustre: Skipped 1 previous similar message [ 4552.700313] Lustre: Skipped 2 previous similar messages [ 4555.548733] Lustre: server umount lustre-MDT0000 complete [ 4559.767105] LustreError: 149932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907478 with bad export cookie 8314361883693097139 [ 4559.768103] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.775668] LustreError: 149932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4560.133197] Lustre: server umount lustre-MDT0001 complete [ 4574.727905] Lustre: server umount lustre-OST0000 complete [ 4595.961790] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4606.598928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4642.257051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.503503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4642.523383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4642.587729] Lustre: lustre-MDT0000: new disk, initializing [ 4642.676653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.707352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4647.261294] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4657.661589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.836582] Lustre: 156720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4657.915432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4657.920700] Lustre: Skipped 1 previous similar message [ 4658.052808] Lustre: lustre-MDT0001: new disk, initializing [ 4658.177988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4658.211442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4658.221243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4662.755636] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4672.348729] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4679.293996] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4687.451504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4687.718230] Lustre: lustre-OST0000: new disk, initializing [ 4687.722352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4687.730380] Lustre: 158694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4687.841493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.679347] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4697.173401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4697.181989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4697.269524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4704.591542] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.472680] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4709.733428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4714.800722] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4715.080723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4718.049734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.054111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.056595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4718.068961] Lustre: Skipped 1 previous similar message [ 4718.090820] LustreError: Skipped 1 previous similar message [ 4723.173606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.190984] Lustre: Skipped 2 previous similar messages [ 4723.538948] Lustre: server umount lustre-OST0000 complete [ 4728.305097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.310072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4728.322124] Lustre: Skipped 1 previous similar message [ 4733.774956] Lustre: server umount lustre-MDT0000 complete [ 4737.586501] LustreError: 157517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907656 with bad export cookie 8314361883693098098 [ 4737.593311] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.598266] LustreError: 157517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4737.928559] Lustre: server umount lustre-MDT0001 complete [ 4741.306808] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:47:38 (1788907658) [ 4742.606853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4744.439107] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:47:41 (1788907661) [ 4750.713215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.083630] LustreError: 161106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.100163] LustreError: 161106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4751.140237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.236152] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4756.456265] LustreError: 161107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.544732] LustreError: 161106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.710836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.420200] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4770.518864] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4774.808595] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4779.784571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.577942] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4789.829540] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4805.604231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.606928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4805.617364] Lustre: Skipped 2 previous similar messages [ 4805.621931] Lustre: Skipped 4 previous similar messages [ 4807.352835] Lustre: server umount lustre-MDT0000 complete [ 4810.160851] LustreError: 161086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907729 with bad export cookie 8314361883693098980 [ 4810.165661] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4810.169107] LustreError: 161086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4810.720640] LustreError: 162200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.841099] LustreError: 163595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.852677] LustreError: 163595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4816.465072] Lustre: server umount lustre-MDT0001 complete [ 4821.111385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4829.351791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.545743] LustreError: 164968:0:(llog.c:1655:llog_backup()) MGC192.168.204.127@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4829.557079] Lustre: 164968:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.127@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4840.418676] LustreError: 164973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.432658] LustreError: 164973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4856.833445] LustreError: 164974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.843213] LustreError: 164974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 4890.591777] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4890.593541] LustreError: 162936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.595730] Lustre: 164968:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4890.616310] LustreError: 162936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 4931.114911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4931.126077] Lustre: Skipped 2 previous similar messages [ 4934.566538] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4935.141754] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4941.194988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.618483] LustreError: 162935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.638840] LustreError: 162935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 46 previous similar messages [ 4959.370178] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4961.255879] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4964.228379] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4965.224354] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4969.554372] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4970.811527] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4986.854726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.855497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.864291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.875721] Lustre: Skipped 2 previous similar messages [ 4986.889180] Lustre: Skipped 8 previous similar messages [ 4988.096179] Lustre: server umount lustre-MDT0000 complete [ 4990.975544] LustreError: 161088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907909 with bad export cookie 8314361883693105441 [ 4990.977651] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4990.987232] LustreError: 161088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4991.250332] Lustre: server umount lustre-MDT0001 complete [ 5004.590426] Lustre: server umount lustre-OST0000 complete [ 5020.141091] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5026.445372] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5048.817172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.965240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5048.991400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5049.035386] Lustre: lustre-MDT0000: new disk, initializing [ 5049.086346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.090322] Lustre: Skipped 1 previous similar message [ 5049.101756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5051.667965] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5058.752430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.836161] Lustre: 170072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5059.019817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5061.635447] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5067.849922] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5071.595846] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5076.136152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5076.304786] Lustre: 172038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5080.464990] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5082.129047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5082.139644] Lustre: Skipped 1 previous similar message [ 5082.147628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5082.207700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5086.978850] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5090.058056] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5090.206412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5092.977513] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5093.126832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5097.440488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.445957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.448673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5097.458042] Lustre: Skipped 2 previous similar messages [ 5097.472299] Lustre: Skipped 1 previous similar message [ 5100.548038] Lustre: server umount lustre-OST0000 complete [ 5105.121509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5105.127780] LustreError: Skipped 1 previous similar message [ 5108.903242] Lustre: server umount lustre-MDT0000 complete [ 5111.237705] LustreError: 170064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908030 with bad export cookie 8314361883693122269 [ 5111.239894] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.244940] LustreError: 170064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5111.457656] Lustre: server umount lustre-MDT0001 complete [ 5117.715729] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:53:54 (1788908034) [ 5122.542812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.905995] LustreError: 174447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.815286] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5131.700563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.589384] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5139.256749] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5143.083612] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5147.556352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5151.345960] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5155.470790] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5178.358531] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:54:55 (1788908095) [ 5182.946360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.952853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.957320] Lustre: Skipped 3 previous similar messages [ 5182.969273] Lustre: Skipped 4 previous similar messages [ 5187.389716] Lustre: server umount lustre-OST0000 complete [ 5196.251674] Lustre: server umount lustre-MDT0000 complete [ 5198.912467] LustreError: 174428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908117 with bad export cookie 8314361883693123151 [ 5198.920311] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.924954] LustreError: 174428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.151978] Lustre: server umount lustre-MDT0001 complete [ 5206.823569] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5208.622096] Key type lgssc unregistered [ 5208.829539] LNet: 178193:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5208.834605] LNetError: 178193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5208.855690] LNet: Removed LNI 192.168.204.127@tcp [ 5209.442080] Key type .llcrypt unregistered [ 5209.445094] Key type ._llcrypt unregistered [ 5222.698541] Key type ._llcrypt registered [ 5222.701497] Key type .llcrypt registered [ 5222.782812] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5223.477192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5223.503433] alg: No test for adler32 (adler32-zlib) [ 5224.489344] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5224.687820] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5226.311780] Key type lgssc registered [ 5227.064231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5232.221627] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5232.241076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.712745] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5252.831341] LustreError: 179415:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 5257.952840] LustreError: 179415:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5257.972644] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5257.977271] LustreError: MGC192.168.204.127@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5257.988420] LustreError: 179415:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5257.998581] LustreError: 179415:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -22 [ 5258.006670] LustreError: 179415:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 5258.060703] Lustre: server umount lustre-OST0000 complete [ 5258.065349] LustreError: 179415:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5264.141569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.441463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.096642] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5273.463339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.678780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.310743] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5281.296581] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5285.287469] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5289.977866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.103633] LustreError: 181729:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5290.111152] Lustre: 181729:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.127@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5290.209515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.966750] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5298.447637] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5303.778101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.787629] Lustre: Skipped 2 previous similar messages [ 5303.788496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5303.795173] Lustre: Skipped 2 previous similar messages [ 5308.898884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.904216] Lustre: Skipped 2 previous similar messages [ 5314.015508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5314.019627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.027362] Lustre: Skipped 2 previous similar messages [ 5314.091199] Lustre: server umount lustre-MDT0000 complete [ 5316.408510] LustreError: 179895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908235 with bad export cookie 6536616846033745851 [ 5316.409911] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.418954] LustreError: 179895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5316.605599] Lustre: server umount lustre-MDT0001 complete [ 5329.682564] Lustre: server umount lustre-OST0000 complete [ 5335.966756] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5343.196141] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5369.196514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.327214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5369.344876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5369.390745] Lustre: lustre-MDT0000: new disk, initializing [ 5369.430807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.447628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5372.395559] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5379.820825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.908357] Lustre: 185254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5379.955449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5379.958490] Lustre: Skipped 1 previous similar message [ 5380.002168] Lustre: lustre-MDT0001: new disk, initializing [ 5380.055735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5380.079792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5380.085095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5382.866704] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5390.075976] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5394.187151] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5398.772105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5398.937031] Lustre: lustre-OST0000: new disk, initializing [ 5398.939335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5398.945586] Lustre: 187223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5398.998483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5402.907534] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5405.724182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5405.732617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5405.808097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5409.529474] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5412.270771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5412.394624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5415.376119] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5415.521517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5421.029694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5421.030041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.035741] LustreError: Skipped 1 previous similar message [ 5421.050226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5421.056719] Lustre: Skipped 1 previous similar message [ 5422.781463] Lustre: server umount lustre-OST0000 complete [ 5426.147475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5426.148611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.156034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.176465] Lustre: Skipped 2 previous similar messages [ 5431.253512] Lustre: server umount lustre-MDT0000 complete [ 5431.264816] LustreError: 185265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5431.276651] LustreError: 185265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5433.528205] LustreError: 185247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908352 with bad export cookie 6536616846033746565 [ 5433.533024] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5433.539058] LustreError: 185247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5433.726428] Lustre: server umount lustre-MDT0001 complete [ 5438.952750] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:59:16 (1788908356) [ 5449.636677] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5451.639920] Key type lgssc unregistered [ 5451.852700] LNet: 190165:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5451.859662] LNetError: 190165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5451.872880] LNet: Removed LNI 192.168.204.127@tcp [ 5452.353177] Key type .llcrypt unregistered [ 5452.355196] Key type ._llcrypt unregistered [ 5463.506581] Key type ._llcrypt registered [ 5463.510294] Key type .llcrypt registered [ 5463.572848] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5464.040670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5464.057893] alg: No test for adler32 (adler32-zlib) [ 5464.993639] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5465.135549] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5466.751204] Key type lgssc registered [ 5467.405811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5472.268177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5472.284552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.584665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.330620] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5481.848656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.064695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5484.579154] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5488.424380] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5491.174901] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5495.422710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5495.541695] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5495.546648] LustreError: 193221:0:(llog.c:1696:llog_backup()) MGC192.168.204.127@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5495.555767] Lustre: 193221:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.127@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5495.633389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.805074] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5502.859427] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5503.969564] LustreError: 191379:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5503.971882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.976138] LustreError: 191379:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5503.984677] Lustre: Skipped 1 previous similar message [ 5503.986529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.997568] Lustre: Skipped 2 previous similar messages [ 5507.041941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.052777] Lustre: Skipped 2 previous similar messages [ 5512.164328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.168404] Lustre: Skipped 2 previous similar messages [ 5517.282317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.288143] Lustre: Skipped 2 previous similar messages [ 5518.303127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5518.395260] Lustre: server umount lustre-MDT0000 complete [ 5520.940044] LustreError: 192092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908439 with bad export cookie 16830761606757871957 [ 5520.941166] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5520.946988] LustreError: 192092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5521.335499] LustreError: 192087:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000066c4ba1f x1875806396637184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5521.447410] Lustre: server umount lustre-MDT0001 complete [ 5533.243556] Lustre: server umount lustre-OST0000 complete [ 5538.449699] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5544.121829] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5564.358532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.506455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5564.530958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5564.582762] Lustre: lustre-MDT0000: new disk, initializing [ 5564.627947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.643764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5567.119905] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5574.502803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.601480] Lustre: 196757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5574.640958] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5574.642861] Lustre: Skipped 1 previous similar message [ 5574.645324] LustreError: 197537:0:(llog.c:1696:llog_backup()) MGC192.168.204.127@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5574.651151] LustreError: 197537:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5574.656821] Lustre: 197537:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.127@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5574.672852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5574.677682] Lustre: Skipped 1 previous similar message [ 5574.750589] Lustre: lustre-MDT0001: new disk, initializing [ 5574.823781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5574.860483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5574.874107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5577.226488] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5582.849980] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5585.984158] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5589.959046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.107178] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5590.109699] Lustre: Skipped 1 previous similar message [ 5590.111647] LustreError: 198697:0:(llog.c:1696:llog_backup()) MGC192.168.204.127@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5590.116267] LustreError: 198697:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5590.120201] Lustre: 198697:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.127@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5590.124750] Lustre: 198697:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5590.137613] Lustre: lustre-OST0000: new disk, initializing [ 5590.141479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5590.144268] Lustre: 198721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5590.194514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.313222] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5596.699559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5596.707866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5596.780696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5598.912928] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5601.510781] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5601.622415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5603.947540] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5604.065076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5606.883375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5606.886885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.890778] LustreError: Skipped 1 previous similar message [ 5606.892222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.898187] Lustre: Skipped 2 previous similar messages [ 5606.911891] Lustre: Skipped 1 previous similar message [ 5611.320205] Lustre: server umount lustre-OST0000 complete [ 5616.096455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.103543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.116140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5619.782850] Lustre: server umount lustre-MDT0000 complete [ 5622.090327] LustreError: 197550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908540 with bad export cookie 16830761606757872678 [ 5622.091087] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.101550] LustreError: 197550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5622.285535] Lustre: server umount lustre-MDT0001 complete [ 5627.614429] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:02:24 (1788908544) [ 5637.827989] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5645.110036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.338816] LustreError: 201846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.349528] LustreError: 201846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5645.360267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.269561] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5650.401985] LustreError: 201847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.520753] LustreError: 201846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.120353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.650387] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5674.483357] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5686.269795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5686.681819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.688566] Lustre: Skipped 1 previous similar message [ 5696.718528] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5708.311618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.492793] Lustre: lustre-OST0001: new disk, initializing [ 5708.496839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5708.503210] Lustre: 204155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5715.364902] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5718.716704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5718.723952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5718.851384] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5727.710410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5734.872803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5737.350303] Lustre: DEBUG MARKER: ADD OST3 [ 5743.443759] loop: module loaded [ 5743.547189] loop0: detected capacity change from 0 to 204800000 [ 5743.785141] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5743.808942] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5743.852060] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5750.469756] loop0: detected capacity change from 0 to 204800000 [ 5751.267514] Lustre: DEBUG MARKER: STOP OST3 [ 5756.906959] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5757.130403] Lustre: lustre-OST0003: new disk, initializing [ 5757.135110] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5757.145497] Lustre: 206175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5757.343351] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5757.347666] Lustre: Skipped 1 previous similar message [ 5758.982043] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5759.196421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5759.205701] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5759.265295] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5760.957421] Lustre: Failing over lustre-OST0003 [ 5760.997768] LustreError: 206331:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5761.162479] Lustre: server umount lustre-OST0003 complete [ 5763.861915] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5764.123527] LustreError: 203336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.593194] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5764.600724] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.608665] Lustre: Skipped 1 previous similar message [ 5769.220039] LustreError: 204641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.242240] LustreError: 204641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5772.473475] Lustre: DEBUG MARKER: START OST3 again [ 5778.680453] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.111167] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5779.478677] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5781.015178] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5781.050575] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5785.844452] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5807.486753] Lustre: Failing over lustre-OST0003 [ 5807.623124] Lustre: server umount lustre-OST0003 complete [ 5810.166660] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.182828] Lustre: Skipped 1 previous similar message [ 5810.184737] LustreError: 203336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.210286] LustreError: 203336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5829.600676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5829.619509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.648235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.663502] Lustre: Skipped 2 previous similar messages [ 5830.638233] LustreError: 203336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.661927] LustreError: 203336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5833.051219] Lustre: server umount lustre-MDT0000 complete [ 5837.010594] LustreError: 201827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908755 with bad export cookie 16830761606757873560 [ 5837.012934] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5837.019344] LustreError: 201827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5843.438873] Lustre: server umount lustre-MDT0001 complete [ 5847.880871] Lustre: server umount lustre-OST0000 complete [ 5851.873533] Lustre: server umount lustre-OST0001 complete [ 5858.781674] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5867.281628] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5906.577545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.843700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5906.876748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5906.956619] Lustre: lustre-MDT0000: new disk, initializing [ 5907.051839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.058212] Lustre: Skipped 1 previous similar message [ 5907.077462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5911.823697] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5923.586032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.773903] Lustre: 211625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.921217] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5923.927330] Lustre: Skipped 1 previous similar message [ 5924.032116] Lustre: lustre-MDT0001: new disk, initializing [ 5924.203430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5924.216794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5929.230248] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5939.683403] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5945.857732] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5952.762808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.084617] Lustre: 213598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5954.630048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5954.837818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5959.950339] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5970.213991] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5975.295824] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5975.586629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5980.710794] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5980.910178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5985.775428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.783216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.795566] LustreError: Skipped 1 previous similar message [ 5985.797109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5985.797115] Lustre: Skipped 8 previous similar messages [ 5985.833132] Lustre: Skipped 6 previous similar messages [ 5989.067978] Lustre: server umount lustre-OST0000 complete [ 5996.004902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.975715] Lustre: server umount lustre-MDT0000 complete [ 6001.122785] LustreError: 211635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.145122] LustreError: 211635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6002.669511] LustreError: 211617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908921 with bad export cookie 16830761606757875856 [ 6002.673469] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.677703] LustreError: 211617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6003.013054] Lustre: server umount lustre-MDT0001 complete [ 6007.503214] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:08:43 (1788908923) [ 6026.657706] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6038.916586] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6078.351676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.572940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6078.606401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6078.610827] Lustre: Skipped 1 previous similar message [ 6078.691668] Lustre: lustre-MDT0000: new disk, initializing [ 6078.695579] Lustre: Skipped 1 previous similar message [ 6078.815350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6078.822518] Lustre: Skipped 1 previous similar message [ 6083.294521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6093.615932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.769220] Lustre: 218208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6094.052410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6098.468690] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6107.838346] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6113.678125] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6120.560229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.815223] Lustre: 220176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6123.047694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6127.778558] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6137.930597] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.711339] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6143.933918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6148.993529] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6149.230729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6153.702042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.703651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6153.722992] Lustre: Skipped 2 previous similar messages [ 6153.724465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.756509] Lustre: Skipped 4 previous similar messages [ 6157.582875] Lustre: server umount lustre-OST0000 complete [ 6167.415972] Lustre: server umount lustre-MDT0000 complete [ 6169.060147] LustreError: 218221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.088287] LustreError: 218221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6170.842873] LustreError: 218200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909089 with bad export cookie 16830761606757876738 [ 6170.844874] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.856974] LustreError: 218200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.338441] Lustre: server umount lustre-MDT0001 complete [ 6180.598366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.153615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.164469] Lustre: Skipped 5 previous similar messages [ 6185.940624] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6194.334041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.958528] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6205.895426] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.496094] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6218.374809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6224.483509] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6231.121859] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.763977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.788556] Lustre: Skipped 4 previous similar messages [ 6241.798989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.815986] LustreError: Skipped 1 previous similar message [ 6247.159638] Lustre: server umount lustre-OST0000 complete [ 6257.627422] Lustre: server umount lustre-MDT0000 complete [ 6260.525347] LustreError: 223129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909179 with bad export cookie 16830761606757877620 [ 6260.531207] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6260.535149] LustreError: 223129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6260.792935] Lustre: server umount lustre-MDT0001 complete [ 6368.337835] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6371.105123] Key type lgssc unregistered [ 6371.392086] LNet: 226057:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6371.398798] LNetError: 226057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6371.431868] LNet: Removed LNI 192.168.204.127@tcp [ 6372.222182] Key type .llcrypt unregistered [ 6372.225401] Key type ._llcrypt unregistered [ 6395.377441] Key type ._llcrypt registered [ 6395.380361] Key type .llcrypt registered [ 6395.485182] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6410.046352] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6411.065612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6411.081383] alg: No test for adler32 (adler32-zlib) [ 6412.135219] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6412.437110] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6414.151514] Key type lgssc registered [ 6415.137345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.743262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.761235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.929830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6444.954341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6445.015491] Lustre: lustre-MDT0000: new disk, initializing [ 6445.064784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.076956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6448.205594] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6455.843851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.925805] Lustre: 229410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6455.959731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6455.962564] Lustre: Skipped 1 previous similar message [ 6456.011309] Lustre: lustre-MDT0001: new disk, initializing [ 6456.089428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6456.130678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6456.142917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.204596] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6466.578721] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6470.735756] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6475.095256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.266539] Lustre: lustre-OST0000: new disk, initializing [ 6475.269297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6475.273807] Lustre: 231375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6475.331962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.496739] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6483.497559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6483.512188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6483.583845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6486.669574] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6490.803719] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6490.975663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6494.634624] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6494.801504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6498.786201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6498.791062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.798364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6498.811426] Lustre: Skipped 1 previous similar message [ 6502.216845] Lustre: server umount lustre-OST0000 complete [ 6507.489864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6507.497240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6507.514318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6507.520842] Lustre: Skipped 1 previous similar message [ 6509.025113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.026104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.857847] Lustre: server umount lustre-MDT0000 complete [ 6513.354476] LustreError: 229400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909432 with bad export cookie 6043303959957905827 [ 6513.358300] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.362552] LustreError: 229400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6513.574492] Lustre: server umount lustre-MDT0001 complete [ 6527.359476] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6534.788473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.178901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.775580] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6540.259421] LustreError: 234307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.279255] LustreError: 234307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6545.051923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.301263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6548.697418] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6550.776188] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6555.905035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6556.250351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.929209] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6568.254699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.379306] Lustre: lustre-OST0001: new disk, initializing [ 6568.382924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6568.387779] Lustre: 236615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6571.322217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6571.328876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6571.403024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6573.211441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6582.622911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6588.315373] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6598.071619] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:18:35 (1788909515) [ 6601.700035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6601.707794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.717105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.719665] Lustre: Skipped 1 previous similar message [ 6605.761715] Lustre: server umount lustre-MDT0000 complete [ 6607.330464] LustreError: 234307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.340364] LustreError: 234307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6608.089366] LustreError: 237442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909526 with bad export cookie 6043303959957906716 [ 6608.089755] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6608.104093] LustreError: 237442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6608.301285] Lustre: server umount lustre-MDT0001 complete [ 6620.705046] Lustre: server umount lustre-OST0000 complete [ 6633.956796] Lustre: server umount lustre-OST0001 complete [ 6639.222645] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6647.880919] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6672.145834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.318161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6672.347942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6672.407927] Lustre: lustre-MDT0000: new disk, initializing [ 6672.482822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.490805] Lustre: Skipped 1 previous similar message [ 6672.505851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6675.595339] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6683.811127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.923435] Lustre: 240555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6683.980225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6683.985495] Lustre: Skipped 1 previous similar message [ 6684.043117] Lustre: lustre-MDT0001: new disk, initializing [ 6684.151515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6684.157390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6687.296588] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6694.062107] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6698.139792] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6702.990157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.181262] Lustre: lustre-OST0000: new disk, initializing [ 6703.185567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6703.191815] Lustre: 242524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6704.282579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6704.290188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6704.375213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6707.447749] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6713.933669] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6716.557432] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6716.670730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6719.219421] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6719.370979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6725.093761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.094546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.098717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.098728] Lustre: Skipped 4 previous similar messages [ 6725.119246] Lustre: Skipped 4 previous similar messages [ 6726.650248] Lustre: server umount lustre-OST0000 complete [ 6730.218099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6730.223290] LustreError: Skipped 1 previous similar message [ 6735.331380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6735.343850] Lustre: Skipped 4 previous similar messages [ 6735.677676] Lustre: server umount lustre-MDT0000 complete [ 6738.272238] LustreError: 240548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909657 with bad export cookie 6043303959957908851 [ 6738.272699] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.281214] LustreError: 240548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6738.488688] Lustre: server umount lustre-MDT0001 complete [ 6744.619556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.926646] LustreError: 244737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.984686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6744.989261] Lustre: Skipped 2 previous similar messages [ 6748.430535] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6750.177927] LustreError: 244738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.274234] LustreError: 244737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.992937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.659951] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6764.207714] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.239090] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6773.372306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.582511] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6784.314793] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6928.353600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6928.358197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.360512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6928.369972] Lustre: Skipped 3 previous similar messages [ 6942.175195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.267455] Lustre: server umount lustre-OST0000 complete [ 6948.834279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.071145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6959.189488] Lustre: server umount lustre-MDT0000 complete [ 6961.439268] LustreError: 244719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909880 with bad export cookie 6043303959957909733 [ 6961.442641] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.448918] LustreError: 244719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6961.655973] Lustre: server umount lustre-MDT0001 complete [ 7049.693568] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7051.341770] Key type lgssc unregistered [ 7051.536866] LNet: 248347:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7051.546554] LNetError: 248347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7051.561019] LNet: Removed LNI 192.168.204.127@tcp [ 7052.101237] Key type .llcrypt unregistered [ 7052.104661] Key type ._llcrypt unregistered [ 7066.607093] Key type ._llcrypt registered [ 7066.608670] Key type .llcrypt registered [ 7066.663564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7076.288648] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7076.961885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.007322] alg: No test for adler32 (adler32-zlib) [ 7077.985721] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7078.172153] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7079.807753] Key type lgssc registered [ 7080.488146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7106.056656] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7111.476476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7111.494565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.636783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7112.659288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7112.709328] Lustre: lustre-MDT0000: new disk, initializing [ 7112.749987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.767101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7115.289709] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7121.479839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.545466] Lustre: 252391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7121.585319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7121.590381] Lustre: Skipped 1 previous similar message [ 7121.651491] Lustre: lustre-MDT0001: new disk, initializing [ 7121.728468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7121.756693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7121.767713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7124.044735] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7127.201355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7131.154329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.360181] Lustre: lustre-OST0000: new disk, initializing [ 7131.363735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7131.370316] Lustre: 254025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7131.439493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.209782] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7138.853704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7138.865756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7138.961903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7144.210062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7144.341626] Lustre: lustre-OST0001: new disk, initializing [ 7144.344652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7144.348359] Lustre: 254895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7144.419443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7146.745791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7146.751098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7146.814478] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7148.246706] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7157.312494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7162.800530] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7170.830475] Lustre: server umount lustre-MDT0000 complete [ 7172.582208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.589424] Lustre: Skipped 2 previous similar messages [ 7173.262083] LustreError: 252381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910092 with bad export cookie 9815967016908710579 [ 7173.269598] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.272331] LustreError: 252381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7175.511756] Lustre: server umount lustre-MDT0001 complete [ 7188.645132] Lustre: server umount lustre-OST0000 complete [ 7201.547109] Lustre: server umount lustre-OST0001 complete [ 7209.399457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7214.877579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7220.123809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7226.636950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7238.897690] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7245.785857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.807407] Lustre: lustre-MDT0000: reset Object Index mappings [ 7246.107510] LustreError: 259897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.129052] LustreError: 259897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7246.171113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.410860] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7251.435668] LustreError: 259898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.521479] LustreError: 259897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.759056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.782253] Lustre: lustre-MDT0001: reset Object Index mappings [ 7256.154955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7259.857967] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7262.016381] Lustre: 261039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7266.819524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.254195] LustreError: 261392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.279872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7271.629479] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7275.494875] LustreError: 261392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.604557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7277.836870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.842598] Lustre: Skipped 1 previous similar message [ 7280.942808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7281.994757] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7287.367989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7289.728994] Lustre: 262908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7296.482372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.490875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7296.492317] Lustre: Skipped 4 previous similar messages [ 7296.501319] Lustre: Skipped 3 previous similar messages [ 7298.745749] Lustre: server umount lustre-MDT0000 complete [ 7301.481524] LustreError: 259877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910220 with bad export cookie 9815967016908717362 [ 7301.484059] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7301.491389] LustreError: 259877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7301.605883] LustreError: 259894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.610978] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.615452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7301.620489] LustreError: 259894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7301.632408] Lustre: Skipped 1 previous similar message [ 7301.741714] Lustre: server umount lustre-MDT0001 complete [ 7310.318644] Lustre: server umount lustre-OST0000 complete [ 7313.202161] Lustre: server umount lustre-OST0001 complete [ 7325.835172] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:30:43 (1788910243) [ 7335.751651] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7342.468612] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7367.441940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.661806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7367.691106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7367.744849] Lustre: lustre-MDT0000: new disk, initializing [ 7367.818353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.831239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7371.818254] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7378.891402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.969883] Lustre: 267160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7379.019552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7379.022977] Lustre: Skipped 1 previous similar message [ 7379.081534] Lustre: lustre-MDT0001: new disk, initializing [ 7379.160135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7379.165911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7381.685864] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7387.169485] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7390.457308] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7394.390816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7394.607298] Lustre: 269129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7396.053405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7396.062440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7396.192561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7398.495603] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7404.441850] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.981214] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7407.079887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7409.403497] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7409.534105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7411.683321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7411.683942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.690392] LustreError: Skipped 1 previous similar message [ 7411.698908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7411.704591] Lustre: Skipped 2 previous similar messages [ 7416.840217] Lustre: server umount lustre-OST0000 complete [ 7420.384367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7420.395611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.407336] Lustre: Skipped 1 previous similar message [ 7420.411418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7427.040633] LustreError: 268056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.061932] LustreError: 268056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7427.382755] Lustre: server umount lustre-MDT0000 complete [ 7429.603533] LustreError: 267956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910348 with bad export cookie 9815967016908719455 [ 7429.606743] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7429.613026] LustreError: 267956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7429.806338] Lustre: server umount lustre-MDT0001 complete [ 7441.391730] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7446.714947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.988435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.992126] Lustre: Skipped 2 previous similar messages [ 7449.057326] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7453.966689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.266779] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7457.679587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7461.180176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.568763] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7469.722202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.855380] Lustre: lustre-OST0001: new disk, initializing [ 7469.858417] Lustre: Skipped 1 previous similar message [ 7469.862754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7469.868522] Lustre: Skipped 1 previous similar message [ 7469.873246] Lustre: 274364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7471.303911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7471.311878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7471.387891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7473.829803] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7479.955839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7482.141718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7485.163753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7485.179136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7490.017056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7490.026994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.033262] Lustre: Skipped 1 previous similar message [ 7490.036587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7490.039583] Lustre: Skipped 2 previous similar messages [ 7495.125174] Lustre: server umount lustre-MDT0000 complete [ 7496.901225] LustreError: 272041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910415 with bad export cookie 9815967016908720330 [ 7496.904454] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7496.906735] LustreError: 272041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7497.044780] Lustre: server umount lustre-MDT0001 complete [ 7509.139822] Lustre: server umount lustre-OST0000 complete [ 7520.456746] Lustre: server umount lustre-OST0001 complete [ 7529.326564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7534.718068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.916099] LustreError: 277414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.922616] LustreError: 277414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7536.724178] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7540.690348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.021484] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7544.460818] Lustre: 278554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7547.654908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.623217] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7554.642977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7558.095138] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7559.921070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7559.940104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7567.926546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7569.794987] Lustre: 280433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7572.340532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7572.353855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7576.792392] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:34:54 (1788910494) [ 7590.886086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.887656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7590.892911] Lustre: Skipped 6 previous similar messages [ 7590.896290] Lustre: Skipped 6 previous similar messages [ 7595.096815] LustreError: 281145:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8ac0031f4d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7597.185824] Lustre: server umount lustre-MDT0000 complete [ 7599.013338] LustreError: 277415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7599.013908] LustreError: 278556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910517 with bad export cookie 9815967016908726651 [ 7599.018358] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7599.023239] LustreError: 277415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7599.028994] LustreError: 278556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7608.291564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7608.296811] Lustre: Skipped 3 previous similar messages [ 7613.407454] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7613.538404] Lustre: server umount lustre-MDT0001 complete [ 7621.658202] Lustre: server umount lustre-OST0000 complete [ 7629.783478] Lustre: server umount lustre-OST0001 complete [ 7633.693143] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7639.720160] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7667.371643] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7673.095217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.233805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7673.255090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7673.305186] Lustre: lustre-MDT0000: new disk, initializing [ 7673.357703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.361857] Lustre: Skipped 7 previous similar messages [ 7673.371208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7675.702405] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7682.014576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.095274] Lustre: 284673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.292653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7684.924656] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7692.141232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.295425] Lustre: 286306:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7694.279122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7695.558812] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7701.808129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7701.901766] Lustre: 287177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7703.272691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7705.136483] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7711.143586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7712.993590] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7712.996318] Lustre: Skipped 1 previous similar message [ 7718.942558] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:37:16 (1788910636) [ 7723.488612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7723.493302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7723.500434] Lustre: Skipped 2 previous similar messages [ 7723.505310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7723.509344] Lustre: Skipped 2 previous similar messages [ 7726.205408] Lustre: server umount lustre-MDT0000 complete [ 7728.203661] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910647 with bad export cookie 9815967016908734533 [ 7728.207866] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7728.209778] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7728.387399] Lustre: server umount lustre-MDT0001 complete [ 7740.445655] Lustre: server umount lustre-OST0000 complete [ 7751.427493] Lustre: server umount lustre-OST0001 complete [ 7755.606183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.846324] LustreError: 289419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.858582] LustreError: 289419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7758.208808] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7762.662063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.087495] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7768.814441] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7771.900435] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7775.224770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7778.608556] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7782.434699] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.267626] loop1: detected capacity change from 0 to 204800000 [ 7801.444871] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7801.456861] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7801.475668] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7805.059811] loop1: detected capacity change from 0 to 204800000 [ 7809.369345] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.431974] LustreError: 289407:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12067: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7809.443653] LustreError: 289407:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12067 [ 7809.449867] LustreError: 289407:0:(mgs_handler.c:517:mgs_target_reg()) fs12067-MDT0001: Error checking target index: rc = -13 [ 7809.458455] LustreError: fs12067-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7809.469292] LustreError: 293155:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 7809.478702] LustreError: 293155:0:(tgt_mount.c:2221:server_put_super()) no obd fs12067-MDT0001 [ 7809.490790] LustreError: 293155:0:(tgt_mount.c:116:server_deregister_mount()) fs12067-MDT0001 not registered [ 7809.527775] Lustre: server umount fs12067-MDT0001 complete [ 7809.530205] LustreError: 293155:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7818.956208] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.057503] Lustre: Setting parameter fs12067-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12067-MDT0001 [ 7819.098168] Lustre: srv-fs12067-MDT0001: No data found on store. Initialize space. [ 7819.102903] Lustre: Skipped 4 previous similar messages [ 7819.179683] Lustre: fs12067-MDT0001: new disk, initializing [ 7819.183618] Lustre: Skipped 3 previous similar messages [ 7819.307396] Lustre: srv-fs12067-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7819.319949] Lustre: 293622:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12067-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7819.332633] LustreError: 293622:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12067-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7823.337850] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7829.890748] Lustre: server umount fs12067-MDT0001 complete [ 7839.155929] loop1: detected capacity change from 0 to 204800000 [ 7842.039154] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.221901] Lustre: 295187:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12067-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7842.227846] LustreError: 295187:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12067-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7844.420474] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7846.853990] Lustre: server umount fs12067-MDT0001 complete [ 7852.561918] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 7852.568534] Lustre: Skipped 3 previous similar messages [ 7855.072723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7855.079067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.089045] Lustre: Skipped 3 previous similar messages [ 7858.627855] Lustre: server umount lustre-MDT0000 complete [ 7863.050540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.145330] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.411869] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7868.392118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7869.456591] loop1: detected capacity change from 0 to 204800000 [ 7872.795698] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.165119] Lustre: 297631:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12067-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7873.178140] LustreError: 297631:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12067-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7873.514220] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7875.868888] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7879.340458] Lustre: server umount fs12067-MDT0001 complete [ 7899.107519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.391179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7911.512481] Lustre: server umount lustre-OST0000 complete [ 7919.690750] Lustre: server umount lustre-MDT0000 complete [ 7921.619386] LustreError: 289400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910840 with bad export cookie 9815967016908737214 [ 7921.621343] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7921.626416] LustreError: 289400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7921.803565] Lustre: server umount lustre-MDT0001 complete [ 7926.888188] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7928.413568] Key type lgssc unregistered [ 7928.625541] LNet: 299975:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7928.631893] LNetError: 299975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7928.641546] LNet: Removed LNI 192.168.204.127@tcp [ 7929.180154] Key type .llcrypt unregistered [ 7929.181858] Key type ._llcrypt unregistered [ 7940.409806] Key type ._llcrypt registered [ 7940.411336] Key type .llcrypt registered [ 7940.473880] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7940.983904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7941.003405] alg: No test for adler32 (adler32-zlib) [ 7941.950310] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7942.092346] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7943.727217] Key type lgssc registered [ 7944.254914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7948.278563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7948.289837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.499320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.806995] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7956.533191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.797882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7959.026854] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7963.003773] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7966.148883] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.824268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7970.041831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.341761] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7975.139461] LustreError: 303050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7975.150334] LustreError: 303050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7977.557931] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.909247] LustreError: 303052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.545827] LustreError: 303052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.558410] LustreError: 303052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7985.161442] LustreError: 303546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.273412] LustreError: 303723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.282341] LustreError: 303723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7992.005991] loop1: detected capacity change from 0 to 204800000 [ 7992.083602] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7992.090713] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7992.099140] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7994.557800] loop1: detected capacity change from 0 to 204800000 [ 7997.383839] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.460421] Lustre: fs5056-OST0000: new disk, initializing [ 7997.463045] Lustre: srv-fs5056-OST0000: No data found on store. Initialize space. [ 7997.467340] Lustre: 304954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7997.503421] Lustre: fs5056-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.512209] LustreError: 303053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8000.520101] LustreError: 303053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8000.631319] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8005.622616] Lustre: server umount fs5056-OST0000 complete [ 8011.920678] loop1: detected capacity change from 0 to 204800000 [ 8014.971254] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8015.088086] Lustre: fs5056-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.380476] LustreError: 303723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.388043] LustreError: 303723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8017.926589] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8022.540463] loop2: detected capacity change from 0 to 204800000 [ 8022.634614] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8022.640851] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8022.649963] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8025.340979] loop2: detected capacity change from 0 to 204800000 [ 8028.251398] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.290880] LustreError: fs5056-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8028.299816] LustreError: fs5056-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8028.303732] LustreError: 307993:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 8028.307711] LustreError: 307993:0:(tgt_mount.c:2221:server_put_super()) no obd fs5056-OST0001 [ 8028.310654] LustreError: 307993:0:(tgt_mount.c:116:server_deregister_mount()) fs5056-OST0001 not registered [ 8028.334743] Lustre: server umount fs5056-OST0001 complete [ 8028.336826] LustreError: 307993:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8033.237426] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.309291] Lustre: fs5056-OST0001: new disk, initializing [ 8033.313881] Lustre: srv-fs5056-OST0001: No data found on store. Initialize space. [ 8033.318049] Lustre: 308544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8033.350095] Lustre: fs5056-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8036.237477] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8040.983484] Lustre: server umount fs5056-OST0000 complete [ 8045.569121] Lustre: server umount fs5056-OST0001 complete [ 8051.714601] LustreError: 303546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.722500] LustreError: 303546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8052.018601] loop1: detected capacity change from 0 to 204800000 [ 8054.918135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.028807] Lustre: fs5056-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.895587] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8058.339926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8061.241875] loop2: detected capacity change from 0 to 204800000 [ 8064.041025] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8067.053251] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8068.581417] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8071.347555] Lustre: server umount fs5056-OST0000 complete [ 8076.131667] Lustre: server umount fs5056-OST0001 complete [ 8082.912502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8082.915859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.920752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.940928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.942070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.951793] Lustre: Skipped 1 previous similar message [ 8088.228367] Lustre: server umount lustre-OST0000 complete [ 8090.080182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8090.083456] LustreError: Skipped 1 previous similar message [ 8090.085329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.091896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.178864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.178946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.186800] Lustre: Skipped 1 previous similar message [ 8096.049198] Lustre: server umount lustre-MDT0000 complete [ 8097.774499] LustreError: 301253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911016 with bad export cookie 9057456677598155205 [ 8097.777736] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8097.780210] LustreError: 301253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8097.920159] Lustre: server umount lustre-MDT0001 complete [ 8102.771020] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8103.976517] Key type lgssc unregistered [ 8104.127963] LNet: 314455:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8104.132562] LNetError: 314455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8104.140686] LNet: Removed LNI 192.168.204.127@tcp [ 8104.539121] Key type .llcrypt unregistered [ 8104.540700] Key type ._llcrypt unregistered [ 8120.543848] Key type ._llcrypt registered [ 8120.545612] Key type .llcrypt registered [ 8120.596797] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8129.580275] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8130.073658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8130.107810] alg: No test for adler32 (adler32-zlib) [ 8131.012726] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8131.121549] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8132.711249] Key type lgssc registered [ 8133.172579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8148.770869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8148.777900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.878242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8149.892345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8149.932681] Lustre: lustre-MDT0000: new disk, initializing [ 8149.964946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8149.975502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8151.705251] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8156.874195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8156.931040] Lustre: 318286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8156.961128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8156.963758] Lustre: Skipped 1 previous similar message [ 8157.005683] Lustre: lustre-MDT0001: new disk, initializing [ 8157.053352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8157.072859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8157.076855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8158.871698] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8163.504264] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8165.945673] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8168.917067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.035423] Lustre: lustre-OST0000: new disk, initializing [ 8169.037978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8169.051184] Lustre: 320255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8169.090480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.458875] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8175.975380] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.137193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8176.142829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8176.206910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8176.946933] Lustre: Mounted lustre-client [ 8177.396315] Lustre: Unmounted lustre-client [ 8182.243275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.246267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.250848] Lustre: Skipped 1 previous similar message [ 8187.364228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.367235] Lustre: Skipped 2 previous similar messages [ 8187.496520] Lustre: server umount lustre-OST0000 complete [ 8192.481680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.482476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.487798] Lustre: Skipped 1 previous similar message [ 8195.296820] Lustre: server umount lustre-MDT0000 complete [ 8196.882517] LustreError: 320254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911115 with bad export cookie 9149393087641191402 [ 8196.887073] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8196.888259] LustreError: 320254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8197.044490] Lustre: server umount lustre-MDT0001 complete [ 8201.481692] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8202.749080] Key type lgssc unregistered [ 8236.704398] LNet: 322414:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8236.708409] LNetError: 322414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8308.390612] LNet: Removed LNI 192.168.204.127@tcp [ 8308.793990] Key type .llcrypt unregistered [ 8308.795634] Key type ._llcrypt unregistered [ 8316.326056] Key type ._llcrypt registered [ 8316.327511] Key type .llcrypt registered [ 8316.367779] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8323.512844] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8324.017627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8324.037622] alg: No test for adler32 (adler32-zlib) [ 8324.915852] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8325.023146] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8326.623178] Key type lgssc registered [ 8327.053521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8338.825971] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:47:36 (1788911256) [ 8341.299677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8341.309315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.414894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8342.427633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8342.465186] Lustre: lustre-MDT0000: new disk, initializing [ 8342.491454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.499920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8344.084930] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8348.954840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.010874] Lustre: 325612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8349.045090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8349.047722] Lustre: Skipped 1 previous similar message [ 8349.091439] Lustre: lustre-MDT0001: new disk, initializing [ 8349.137127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8349.155605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8349.160061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8350.699963] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8354.787406] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8356.673425] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8359.202594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8359.311213] Lustre: lustre-OST0000: new disk, initializing [ 8359.312812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8359.316370] Lustre: 327580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8359.351219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.318236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8360.324890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8360.377042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8361.536441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8365.370887] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8375.777029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8375.778279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.779431] LustreError: Skipped 1 previous similar message [ 8375.783513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8375.786870] Lustre: Skipped 1 previous similar message [ 8377.945169] Lustre: server umount lustre-OST0000 complete [ 8379.871966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8379.875857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.881633] Lustre: Skipped 1 previous similar message [ 8379.884201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.897647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.898116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.447919] Lustre: server umount lustre-MDT0000 complete [ 8386.909820] LustreError: 325605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911305 with bad export cookie 15541963942941750957 [ 8386.914169] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8386.915971] LustreError: 325605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8387.060490] Lustre: server umount lustre-MDT0001 complete [ 8391.022356] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8392.176619] Key type lgssc unregistered [ 8392.317036] LNet: 329486:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8392.320481] LNetError: 329486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8392.328338] LNet: Removed LNI 192.168.204.127@tcp [ 8392.692331] Key type .llcrypt unregistered [ 8392.693503] Key type ._llcrypt unregistered [ 8401.857615] Key type ._llcrypt registered [ 8401.858628] Key type .llcrypt registered [ 8401.906686] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8402.269552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8402.328744] alg: No test for adler32 (adler32-zlib) [ 8403.212940] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8403.321455] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8404.927176] Key type lgssc registered [ 8405.346958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8408.356383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8408.365712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.557136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.041722] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8414.444979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.581025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8416.116617] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8419.055192] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8421.186730] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8423.948041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.073983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.274939] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8429.015942] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8438.321720] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8438.979570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8444.898854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.900353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8444.904341] Lustre: Skipped 2 previous similar messages [ 8444.908112] Lustre: Skipped 2 previous similar messages [ 8446.675697] Lustre: server umount lustre-MDT0000 complete [ 8447.956534] LustreError: 330716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911366 with bad export cookie 9912374222490983922 [ 8447.959485] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8447.960842] LustreError: 330716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8450.016459] LustreError: 330732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.016500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.024019] LustreError: 330732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8450.031066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8450.036485] Lustre: Skipped 1 previous similar message [ 8452.064138] LustreError: 330762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.064177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8452.071759] LustreError: 330762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8452.073794] Lustre: Skipped 1 previous similar message [ 8455.166789] LustreError: 332736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.167759] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 8457.183991] LustreError: 330732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.286638] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 8460.290056] Lustre: Skipped 1 previous similar message [ 8462.303120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8462.304251] LustreError: 330732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.313461] LustreError: 330732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8462.384702] Lustre: server umount lustre-MDT0001 complete [ 8469.868582] Lustre: server umount lustre-OST0000 complete [ 8473.588778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.439362] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8476.655890] Lustre: 334916:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.127@tcp [ 8477.646763] Lustre: server umount MGS complete [ 8481.658322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.880064] LustreError: 335487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.920864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8483.677052] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8487.279354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8487.393084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8487.395779] Lustre: Skipped 1 previous similar message [ 8487.454933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8489.103667] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8492.371242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8494.587189] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8497.283647] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.881566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8500.057699] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8512.145743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec [ 8517.090713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.090895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8517.091727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.095104] Lustre: Skipped 1 previous similar message [ 8519.572694] Lustre: server umount lustre-OST0000 complete [ 8522.208593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8522.214655] Lustre: Skipped 1 previous similar message [ 8527.127339] Lustre: server umount lustre-MDT0000 complete [ 8527.329329] LustreError: 336785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.338823] LustreError: 336785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8528.449237] LustreError: 336942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911447 with bad export cookie 9912374222490984748 [ 8528.452015] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8528.453776] LustreError: 336942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8528.569249] Lustre: server umount lustre-MDT0001 complete [ 8532.246807] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8533.276937] Key type lgssc unregistered [ 8533.398317] LNet: 338943:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8533.400270] LNetError: 338943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8533.408285] LNet: Removed LNI 192.168.204.127@tcp [ 8533.694129] Key type .llcrypt unregistered [ 8533.695688] Key type ._llcrypt unregistered [ 8539.845514] Key type ._llcrypt registered [ 8539.846802] Key type .llcrypt registered [ 8539.885150] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8546.618121] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8546.931928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8546.960048] alg: No test for adler32 (adler32-zlib) [ 8547.841802] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8547.947376] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8549.543233] Key type lgssc registered [ 8549.907293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.633125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8562.640152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.734478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8563.746795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8563.786366] Lustre: lustre-MDT0000: new disk, initializing [ 8563.823494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.834878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8565.411300] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8570.099216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.169458] Lustre: 342074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8570.197992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8570.200890] Lustre: Skipped 1 previous similar message [ 8570.238877] Lustre: lustre-MDT0001: new disk, initializing [ 8570.278471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8570.294892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8570.298171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8571.814220] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8575.752929] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8577.799754] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8580.138175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.237480] Lustre: lustre-OST0000: new disk, initializing [ 8580.239756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8580.242767] Lustre: 344042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8580.273686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.403218] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8586.245617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8586.250983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8586.294814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8586.309890] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8588.055825] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8588.142594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8589.705171] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8589.797237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8591.330480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8591.330816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.341195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.343432] Lustre: Skipped 1 previous similar message [ 8592.598839] Lustre: server umount lustre-OST0000 complete [ 8595.935931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.938714] LustreError: Skipped 1 previous similar message [ 8595.941121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.946694] Lustre: Skipped 1 previous similar message [ 8595.949199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8600.159610] Lustre: server umount lustre-MDT0000 complete [ 8601.501744] LustreError: 342067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911520 with bad export cookie 6027840559283601955 [ 8601.506373] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.506580] LustreError: 342067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8601.645941] Lustre: server umount lustre-MDT0001 complete [ 8604.620909] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:52:02 (1788911522) [ 8610.932235] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8615.175826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8629.486923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.583214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8629.596162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8629.634743] Lustre: lustre-MDT0000: new disk, initializing [ 8629.672598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.682907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8631.289783] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8635.831976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.881939] Lustre: 348791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8635.912589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8635.915178] Lustre: Skipped 1 previous similar message [ 8635.953850] Lustre: lustre-MDT0001: new disk, initializing [ 8635.990359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8636.004615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8636.008560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8637.523528] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8641.658748] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8643.770133] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8646.530665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8646.661365] Lustre: lustre-OST0000: new disk, initializing [ 8646.663841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8646.667706] Lustre: 350762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8646.712554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8649.170312] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8653.399091] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8656.376756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8656.379695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8656.413831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8671.714497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8671.716119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.718036] LustreError: Skipped 1 previous similar message [ 8671.725152] Lustre: Skipped 1 previous similar message [ 8671.728665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.732688] Lustre: Skipped 3 previous similar messages [ 8674.137138] Lustre: server umount lustre-OST0000 complete [ 8676.832764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.833124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8676.833541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.836794] Lustre: Skipped 1 previous similar message [ 8681.896663] Lustre: server umount lustre-MDT0000 complete [ 8683.288305] LustreError: 349587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911602 with bad export cookie 6027840559283602823 [ 8683.289553] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8683.293029] LustreError: 349587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8683.425657] Lustre: server umount lustre-MDT0001 complete [ 8687.319589] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8688.309781] Key type lgssc unregistered [ 8688.435412] LNet: 352911:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.438662] LNetError: 352911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8688.446473] LNet: Removed LNI 192.168.204.127@tcp [ 8688.779236] Key type .llcrypt unregistered [ 8688.780685] Key type ._llcrypt unregistered [ 8723.055847] Key type ._llcrypt registered [ 8723.057123] Key type .llcrypt registered [ 8723.094691] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8723.216564] Key type .llcrypt unregistered [ 8723.218129] Key type ._llcrypt unregistered [ 8751.070545] Key type ._llcrypt registered [ 8751.072190] Key type .llcrypt registered [ 8751.111137] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8751.226295] Key type .llcrypt unregistered [ 8751.227749] Key type ._llcrypt unregistered [ 8762.614921] Key type ._llcrypt registered [ 8762.616634] Key type .llcrypt registered [ 8762.656138] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8762.767546] Key type .llcrypt unregistered [ 8762.768717] Key type ._llcrypt unregistered [ 8774.285911] Key type ._llcrypt registered [ 8774.287198] Key type .llcrypt registered [ 8774.328027] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8774.439387] Key type .llcrypt unregistered [ 8774.440814] Key type ._llcrypt unregistered [ 8802.304157] Key type ._llcrypt registered [ 8802.305063] Key type .llcrypt registered [ 8802.344750] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8802.460327] Key type .llcrypt unregistered [ 8802.462107] Key type ._llcrypt unregistered [ 8814.006375] Key type ._llcrypt registered [ 8814.009292] Key type .llcrypt registered [ 8814.055183] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8814.154425] Key type .llcrypt unregistered [ 8814.155876] Key type ._llcrypt unregistered [ 8826.485779] Key type ._llcrypt registered [ 8826.486926] Key type .llcrypt registered [ 8826.532973] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8826.636579] Key type .llcrypt unregistered [ 8826.638252] Key type ._llcrypt unregistered [ 8837.807073] Key type ._llcrypt registered [ 8837.808420] Key type .llcrypt registered [ 8837.846666] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8837.951248] Key type .llcrypt unregistered [ 8837.952330] Key type ._llcrypt unregistered [ 8849.057573] Key type ._llcrypt registered [ 8849.058912] Key type .llcrypt registered [ 8849.097414] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8849.198543] Key type .llcrypt unregistered [ 8849.199834] Key type ._llcrypt unregistered [ 8861.099773] Key type ._llcrypt registered [ 8861.102132] Key type .llcrypt registered [ 8861.154604] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8861.286295] Key type .llcrypt unregistered [ 8861.288010] Key type ._llcrypt unregistered [ 8884.915369] Key type ._llcrypt registered [ 8884.916431] Key type .llcrypt registered [ 8884.958539] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8885.073285] Key type .llcrypt unregistered [ 8885.074641] Key type ._llcrypt unregistered [ 8897.221461] Key type ._llcrypt registered [ 8897.222714] Key type .llcrypt registered [ 8897.267288] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8897.388322] Key type .llcrypt unregistered [ 8897.390044] Key type ._llcrypt unregistered [ 8909.670324] Key type ._llcrypt registered [ 8909.671413] Key type .llcrypt registered [ 8909.710628] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8909.819546] Key type .llcrypt unregistered [ 8909.820881] Key type ._llcrypt unregistered [ 8931.744958] Key type ._llcrypt registered [ 8931.747644] Key type .llcrypt registered [ 8931.794372] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8939.058163] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8939.387893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8939.400177] alg: No test for adler32 (adler32-zlib) [ 8940.256373] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8940.353585] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8941.935181] Key type lgssc registered [ 8942.339245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8954.479713] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8954.484865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.565238] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8955.575853] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8955.612773] Lustre: tst250fs-MDT0000: new disk, initializing [ 8955.633586] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.640024] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.042748] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8961.292282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.329659] Lustre: 372814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8961.352811] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8961.354532] Lustre: Skipped 1 previous similar message [ 8961.389924] Lustre: tst250fs-MDT0001: new disk, initializing [ 8961.419079] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8961.428955] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8961.431645] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8962.851109] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8965.182511] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8967.457845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8967.576889] Lustre: tst250fs-OST0000: new disk, initializing [ 8967.579966] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8967.585428] Lustre: 374446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8967.629861] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.733792] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8972.283320] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8972.287669] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8972.333606] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8974.084803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.159681] Lustre: tst250fs-OST0001: new disk, initializing [ 8974.161776] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8974.165295] Lustre: 375318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8974.204474] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8976.353956] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8979.080398] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8980.633771] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8980.637532] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8980.693141] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8991.202808] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.203986] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.208984] Lustre: Skipped 3 previous similar messages [ 8991.211064] Lustre: Skipped 2 previous similar messages [ 8995.744749] Lustre: server umount tst250fs-MDT0000 complete [ 8997.012216] LustreError: 372807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911915 with bad export cookie 2676231018696807044 [ 8997.015522] LustreError: 372807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8997.015996] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8997.142422] Lustre: server umount tst250fs-MDT0001 complete [ 9008.645672] Lustre: server umount tst250fs-OST0000 complete [ 9019.234802] Lustre: server umount tst250fs-OST0001 complete [ 9025.730833] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9028.790862] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9040.354812] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:59:18 (1788911958) [ 9040.962042] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9046.967345] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9050.783458] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9062.328074] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8720 sec ========== 19:59:40 (1788911980) [ 9063.000674] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:59:40 (1788911980) === [ 9064.259511] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:59:41 (1788911981) === [ 9077.934480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9078.966604] Key type lgssc unregistered [ 9079.089632] LNet: 384446:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9079.094962] LNetError: 384446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9079.105465] LNet: Removed LNI 192.168.204.127@tcp [ 9079.439930] Key type .llcrypt unregistered [ 9079.441823] Key type ._llcrypt unregistered