[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492325427 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003282] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010007] pid_max: default: 32768 minimum: 301 [ 0.011124] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014082] *** VALIDATE selinux *** [ 0.022456] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026807] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] *** VALIDATE tmpfs *** [ 0.030462] *** VALIDATE proc *** [ 0.032192] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035155] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041972] debug: unmapping init [mem 0xffffffffb8459000-0xffffffffb8460fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045641] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053260] rcu: Hierarchical SRCU implementation. [ 0.055430] smp: Bringing up secondary CPUs ... [ 0.056578] x86: Booting SMP configuration: [ 0.057036] .... node #0, CPUs: #1 #2 #3 [ 0.060796] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143171] node 0 deferred pages initialised in 74ms [ 0.146334] devtmpfs: initialized [ 0.147217] x86/mm: Memory block size: 128MB [ 0.149835] gcov: version magic: 0x41383552 [ 0.151234] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153220] pinctrl core: initialized pinctrl subsystem [ 0.154192] [ 0.154871] ************************************************************* [ 0.155015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.157013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158015] ** ** [ 0.159012] ** This means that this kernel is built to expose internal ** [ 0.160013] ** IOMMU data structures, which may compromise security on ** [ 0.161012] ** your system. ** [ 0.162013] ** ** [ 0.163013] ** If you see this message and you are not debugging the ** [ 0.164014] ** kernel, report this immediately to your vendor! ** [ 0.165017] ** ** [ 0.166015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167013] ************************************************************* [ 0.168691] NET: Registered protocol family 16 [ 0.169430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173102] cpuidle: using governor menu [ 0.174729] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175421] PCI: Using configuration type 1 for base access [ 0.176125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185090] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187091] cryptd: max_cpu_qlen set to 1000 [ 0.189239] ACPI: Added _OSI(Module Device) [ 0.190014] ACPI: Added _OSI(Processor Device) [ 0.191012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197470] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205393] ACPI: Interpreter enabled [ 0.206084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207017] ACPI: Using IOAPIC for interrupt routing [ 0.209144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224463] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.237285] acpiphp: Slot [2] registered [ 0.239179] acpiphp: Slot [5] registered [ 0.241169] acpiphp: Slot [6] registered [ 0.243288] acpiphp: Slot [7] registered [ 0.244180] acpiphp: Slot [8] registered [ 0.246188] acpiphp: Slot [9] registered [ 0.247168] acpiphp: Slot [10] registered [ 0.249167] acpiphp: Slot [3] registered [ 0.251122] acpiphp: Slot [4] registered [ 0.253134] acpiphp: Slot [11] registered [ 0.255121] acpiphp: Slot [12] registered [ 0.256109] acpiphp: Slot [13] registered [ 0.258096] acpiphp: Slot [14] registered [ 0.259144] acpiphp: Slot [15] registered [ 0.261096] acpiphp: Slot [16] registered [ 0.262096] acpiphp: Slot [17] registered [ 0.263117] acpiphp: Slot [18] registered [ 0.265099] acpiphp: Slot [19] registered [ 0.266097] acpiphp: Slot [20] registered [ 0.268105] acpiphp: Slot [21] registered [ 0.269102] acpiphp: Slot [22] registered [ 0.271093] acpiphp: Slot [23] registered [ 0.272120] acpiphp: Slot [24] registered [ 0.274093] acpiphp: Slot [25] registered [ 0.275087] acpiphp: Slot [26] registered [ 0.276081] acpiphp: Slot [27] registered [ 0.277104] acpiphp: Slot [28] registered [ 0.279091] acpiphp: Slot [29] registered [ 0.280107] acpiphp: Slot [30] registered [ 0.281086] acpiphp: Slot [31] registered [ 0.282076] PCI host bridge to bus 0000:00 [ 0.282938] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.292018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.295936] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.298327] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313571] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324683] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327787] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331090] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367699] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424248] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.442016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.464018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478088] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486159] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525014] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.535015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.542016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.578398] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.587016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.611023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.620846] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.627016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.633015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.668955] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.671369] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.673463] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.676365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679354] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.684106] iommu: Default domain type: Passthrough [ 0.686425] SCSI subsystem initialized [ 0.688125] ACPI: bus type USB registered [ 0.689114] usbcore: registered new interface driver usbfs [ 0.691098] usbcore: registered new interface driver hub [ 0.693086] usbcore: registered new device driver usb [ 0.695151] pps_core: LinuxPPS API ver. 1 registered [ 0.697013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700079] PTP clock support registered [ 0.704046] EDAC MC: Ver: 3.0.0 [ 0.705485] PCI: Using ACPI for IRQ routing [ 0.706581] NetLabel: Initializing [ 0.707000] NetLabel: domain hash size = 128 [ 0.710016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712080] NetLabel: unlabeled traffic allowed by default [ 0.714186] vgaarb: loaded [ 0.716204] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.717011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.724000] clocksource: Switched to clocksource kvm-clock [ 0.834210] VFS: Disk quotas dquot_6.6.0 [ 0.835948] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.838461] *** VALIDATE ramfs *** [ 0.839628] *** VALIDATE hugetlbfs *** [ 0.841093] pnp: PnP ACPI init [ 0.843260] pnp: PnP ACPI: found 6 devices [ 0.862965] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866078] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.867966] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869927] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.872024] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874168] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876527] NET: Registered protocol family 2 [ 0.878773] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.883376] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.887311] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.892632] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.896211] TCP: Hash tables configured (established 65536 bind 65536) [ 0.899112] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.902138] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904897] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907651] NET: Registered protocol family 1 [ 0.910224] RPC: Registered named UNIX socket transport module. [ 0.912207] RPC: Registered udp transport module. [ 0.914257] RPC: Registered tcp transport module. [ 0.915802] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.918061] NET: Registered protocol family 44 [ 0.919572] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921447] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923526] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.925666] PCI: CLS 0 bytes, default 64 [ 0.928143] Unpacking initramfs... [ 2.332696] debug: unmapping init [mem 0xffff9d167cc54000-0xffff9d167ffbffff] [ 2.336764] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.339326] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.342451] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.847351] Initialise system trusted keyrings [ 2.849465] Key type blacklist registered [ 2.851515] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.861611] zbud: loaded [ 2.865086] *** VALIDATE nfs *** [ 2.866633] *** VALIDATE nfs4 *** [ 2.868478] pstore: using deflate compression [ 2.873158] Platform Keyring initialized [ 2.985589] NET: Registered protocol family 38 [ 2.988039] Key type asymmetric registered [ 2.989639] Asymmetric key parser 'x509' registered [ 2.991659] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.995200] io scheduler mq-deadline registered [ 2.997090] io scheduler kyber registered [ 2.998480] io scheduler bfq registered [ 3.001069] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.004464] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.007489] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.010753] ACPI: Power Button [PWRF] [ 3.016817] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.023251] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.051312] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.058167] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.081279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.110635] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.141797] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.147525] Non-volatile memory driver v1.3 [ 3.149514] Linux agpgart interface v0.103 [ 3.181531] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.185541] vda: detected capacity change from 0 to 75235328 [ 3.201483] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.204529] vdb: detected capacity change from 0 to 1073741824 [ 3.223628] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.226577] vdc: detected capacity change from 0 to 2621440000 [ 3.244272] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.247138] vdd: detected capacity change from 0 to 2621440000 [ 3.260382] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263444] vde: detected capacity change from 0 to 4294967296 [ 3.278372] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.281235] vdf: detected capacity change from 0 to 4294967296 [ 3.288990] libphy: Fixed MDIO Bus: probed [ 3.295427] usbcore: registered new interface driver usbserial_generic [ 3.297773] usbserial: USB Serial support registered for generic [ 3.299513] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.303060] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304510] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.307583] mousedev: PS/2 mouse device common for all mice [ 3.310660] rtc_cmos 00:05: RTC can wake from S4 [ 3.314177] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.314806] rtc_cmos 00:05: registered as rtc0 [ 3.320135] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.322958] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.323692] intel_pstate: CPU model not supported [ 3.329199] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.335361] hid: raw HID events driver (C) Jiri Kosina [ 3.337540] usbcore: registered new interface driver usbhid [ 3.339632] usbhid: USB HID core driver [ 3.341318] drop_monitor: Initializing network drop monitor service [ 3.343986] Initializing XFRM netlink socket [ 3.345971] NET: Registered protocol family 10 [ 3.350971] Segment Routing with IPv6 [ 3.352607] NET: Registered protocol family 17 [ 3.355096] mpls_gso: MPLS GSO support [ 3.361051] RAS: Correctable Errors collector initialized. [ 3.363500] AVX version of gcm_enc/dec engaged. [ 3.365367] AES CTR mode by8 optimization enabled [ 3.448916] sched_clock: Marking stable (3448893666, 0)->(4428316515, -979422849) [ 3.452950] registered taskstats version 1 [ 3.454992] Loading compiled-in X.509 certificates [ 3.457499] zswap: loaded using pool lzo/zbud [ 3.483027] Key type big_key registered [ 3.496676] Key type encrypted registered [ 3.498963] ima: No TPM chip found, activating TPM-bypass! [ 3.501845] ima: Allocated hash algorithm: sha1 [ 3.503314] ima: No architecture policies found [ 3.504992] evm: Initialising EVM extended attributes: [ 3.506819] evm: security.selinux [ 3.508076] evm: security.ima [ 3.509315] evm: security.capability [ 3.510734] evm: HMAC attrs: 0x1 [ 3.513291] rtc_cmos 00:05: setting system clock to 2026-08-27 14:33:01 UTC (1787841181) [ 3.519395] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 3.524159] debug: unmapping init [mem 0xffffffffb8182000-0xffffffffb8458fff] [ 3.532140] Write protecting the kernel read-only data: 28672k [ 3.535328] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 3.538263] debug: unmapping init [mem 0xffffffffb7114000-0xffffffffb71fffff] [ 3.574190] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.583394] systemd[1]: Detected virtualization kvm. [ 3.585524] systemd[1]: Detected architecture x86-64. [ 3.587681] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.620651] systemd[1]: No hostname configured. [ 3.622204] systemd[1]: Set hostname to . [ 3.624115] random: systemd: uninitialized urandom read (16 bytes read) [ 3.626317] systemd[1]: Initializing machine ID from random generator. [ 3.676850] random: ln: uninitialized urandom read (6 bytes read) [ 3.765308] random: systemd: uninitialized urandom read (16 bytes read) [ 3.768133] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.773492] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.777950] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.369927] device-mapper: uevent: version 1.0.3 [ 4.372253] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.122878] random: fast init done [ 5.176233] virtio_net virtio0 ens2: renamed from eth0 [ 5.215959] scsi host0: ata_piix [ 5.248371] scsi host1: ata_piix [ 5.250238] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.252922] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.016891] random: crng init done [ 10.019407] random: 7 urandom warning(s) missed due to ratelimiting [ 10.044587] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.152442] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.762930] printk: systemd: 23 output lines suppressed due to ratelimiting [ 15.453685] SELinux: Disabled at runtime. [ 15.551977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.566884] systemd[1]: Detected virtualization kvm. [ 15.570440] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.180314] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.187703] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.196510] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.206504] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.215203] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.232411] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.255326] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 17.501507] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 18.401067] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.294571] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.600329] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.220797] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.287092] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.710382] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.965390] NFS: Registering the id_resolver key type [ 25.969184] Key type id_resolver registered [ 25.981053] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg618-server login: [ 86.809744] libcfs: loading out-of-tree module taints kernel. [ 86.860691] Key type ._llcrypt registered [ 86.863292] Key type .llcrypt registered [ 86.994944] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 104.790129] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 106.229969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.252703] alg: No test for adler32 (adler32-zlib) [ 107.546100] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 108.439706] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 110.255191] Key type lgssc registered [ 112.007616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.677700] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.164026] hrtimer: interrupt took 4995865 ns [ 174.508358] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 188.220610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 188.262288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.461409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.489613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.555987] Lustre: lustre-MDT0000: new disk, initializing [ 189.634510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.653213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.937098] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.804498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.928911] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.962926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.967934] Lustre: Skipped 1 previous similar message [ 207.069994] Lustre: lustre-MDT0001: new disk, initializing [ 207.143380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.175801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.190948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.848972] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.082091] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.089789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.354790] Lustre: lustre-OST0000: new disk, initializing [ 224.359463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.382839] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.468181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.493400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.503906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.582431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.293489] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.964707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.142902] Lustre: lustre-OST0001: new disk, initializing [ 246.147616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 246.155382] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.229268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.408990] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.008322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 254.019888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 254.097395] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.671728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.459148] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.448447] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.309550] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing yml_node [ 285.154319] Lustre: DEBUG MARKER: Client: 2.17.57.95 [ 287.694584] Lustre: DEBUG MARKER: MDS: 2.17.57.95 [ 290.097832] Lustre: DEBUG MARKER: OSS: 2.17.57.95 [ 291.571150] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:37:47 EDT 2026 [ 307.443658] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 308.914731] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.088104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 325.100170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.122792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.672584] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.675023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.707503] Lustre: Skipped 2 previous similar messages [ 330.487475] Lustre: server umount lustre-MDT0000 complete [ 334.240662] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841512 with bad export cookie 7088900450316160851 [ 334.241886] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.252951] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 338.913078] LustreError: 8946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.921852] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.925170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.949085] LustreError: 8946:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 338.965297] Lustre: Skipped 3 previous similar messages [ 339.936197] LustreError: 8946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.965848] LustreError: 8946:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 340.691770] Lustre: server umount lustre-MDT0001 complete [ 344.675833] Lustre: server umount lustre-OST0000 complete [ 349.415631] Lustre: server umount lustre-OST0001 complete [ 358.553925] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 369.549653] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 406.873575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.088345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 407.110662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.177534] Lustre: lustre-MDT0000: new disk, initializing [ 407.261025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.281785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.935162] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 422.833056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.992666] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.083046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.087273] Lustre: Skipped 1 previous similar message [ 423.145850] Lustre: lustre-MDT0001: new disk, initializing [ 423.223272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.255117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.260772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.810751] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 437.587059] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 443.818118] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.780215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.093609] Lustre: lustre-OST0000: new disk, initializing [ 451.097763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.103080] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.226162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.744251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.752987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.906889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.717770] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 466.679358] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 471.234251] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.426904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.308846] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.467824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.690373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.692598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.706879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.706886] Lustre: Skipped 3 previous similar messages [ 478.735209] Lustre: Skipped 1 previous similar message [ 483.401836] Lustre: server umount lustre-OST0000 complete [ 488.928761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.939622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 488.949667] Lustre: Skipped 2 previous similar messages [ 493.084702] Lustre: server umount lustre-MDT0000 complete [ 494.056710] LustreError: 15093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.096511] LustreError: 15093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 496.811920] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841674 with bad export cookie 7088900450316162384 [ 496.816973] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.817505] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 497.273850] Lustre: server umount lustre-MDT0001 complete [ 503.356328] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:41:18 (1787841678) [ 521.674587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.281092] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.381539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.718848] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 527.843443] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.886486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.140753] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 547.920630] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 554.525449] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 556.781946] Lustre: Failing over lustre-MDT0000 [ 557.140316] Lustre: server umount lustre-MDT0000 complete [ 558.051693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.058678] LustreError: 20652:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.068527] Lustre: Skipped 2 previous similar messages [ 558.081287] LustreError: 20652:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 574.432852] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841736/real 1787841736] req@ffff9d15c7e76d00 x1874687464466176/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 574.461840] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.473248] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.494017] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 576.613255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.640535] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 590.306275] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 590.314180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 590.347638] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.670974] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 604.643109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.656508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.661986] Lustre: Skipped 1 previous similar message [ 604.676028] Lustre: Skipped 1 previous similar message [ 610.686325] Lustre: server umount lustre-MDT0000 complete [ 610.789354] LustreError: 19921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.804874] LustreError: 19921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 614.574439] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841792 with bad export cookie 7088900450316163798 [ 614.581883] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.583107] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 614.978185] Lustre: server umount lustre-MDT0001 complete [ 623.229574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.793639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.802625] Lustre: Skipped 2 previous similar messages [ 629.067577] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 637.465856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.960613] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 650.355152] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.359805] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 658.497225] Lustre: Failing over lustre-MDT0000 [ 658.925612] Lustre: server umount lustre-MDT0000 complete [ 662.636265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.662461] Lustre: Skipped 1 previous similar message [ 678.382240] LustreError: 24309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.411190] LustreError: 24309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 678.660275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.735128] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841840/real 1787841840] req@ffff9d15c849cb40 x1874687464496000/t0(0) o400->MGC192.168.206.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 678.758560] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.429759] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 693.527310] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 693.729427] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 693.739199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 693.753469] Lustre: Skipped 1 previous similar message [ 693.774760] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 703.954705] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 709.096173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.110110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.118094] Lustre: Skipped 2 previous similar messages [ 722.399141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 722.679493] Lustre: server umount lustre-MDT0000 complete [ 727.061924] LustreError: 23242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841905 with bad export cookie 7088900450316164736 [ 727.080279] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.825086] Lustre: server umount lustre-MDT0001 complete [ 736.647260] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:45:12 (1787841912) [ 749.725643] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 759.366542] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 797.276248] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 808.694872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.078963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 809.104472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 809.300148] Lustre: lustre-MDT0000: new disk, initializing [ 809.427804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.432854] Lustre: Skipped 2 previous similar messages [ 809.450817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 814.883865] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 826.438383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.610715] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 826.952759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 831.549498] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 841.872616] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 847.727735] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 854.107416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 854.372674] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 856.006648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 856.040900] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 856.115334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 856.134459] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 860.303705] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 870.367414] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 876.249332] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 897.002791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.006715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.007065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.007070] Lustre: Skipped 6 previous similar messages [ 897.027457] Lustre: Skipped 3 previous similar messages [ 897.047662] LustreError: Skipped 1 previous similar message [ 901.031591] Lustre: server umount lustre-OST0000 complete [ 911.255359] Lustre: server umount lustre-MDT0000 complete [ 912.356556] LustreError: 32206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.378152] LustreError: 32206:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 914.894442] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842092 with bad export cookie 7088900450316165142 [ 914.899964] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.905295] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 915.292857] Lustre: server umount lustre-MDT0001 complete [ 927.319486] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 930.075852] Key type lgssc unregistered [ 930.393680] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.404763] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.429643] LNet: Removed LNI 192.168.206.118@tcp [ 932.378181] Key type .llcrypt unregistered [ 932.382780] Key type ._llcrypt unregistered [ 955.843239] Key type ._llcrypt registered [ 955.847593] Key type .llcrypt registered [ 955.940884] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 957.134470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.146043] alg: No test for adler32 (adler32-zlib) [ 958.133161] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 958.344722] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 959.983175] Key type lgssc registered [ 960.932536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 967.382556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.409230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.817392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.773796] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 978.403558] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.424237] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 980.981906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.502915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 985.248837] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 991.671161] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.543697] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.789450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1003.079885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.199293] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1008.624072] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1016.168932] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.531817] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:50:04 (1787842204) [ 1030.324561] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1032.109104] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1038.556570] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:50:14 (1787842214) [ 1047.222729] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:50:22 (1787842222) [ 1048.184314] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1056.471059] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1056.477418] Lustre: Skipped 2 previous similar messages [ 1058.152691] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:50:33 (1787842233) [ 1061.606248] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1061.610027] Lustre: Skipped 3 previous similar messages [ 1075.570405] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1075.580569] Lustre: Skipped 6 previous similar messages [ 1078.044945] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:50:53 (1787842253) [ 1080.520568] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1083.491595] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1090.870909] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:51:06 (1787842266) [ 1092.401608] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1092.407218] Lustre: Skipped 3 previous similar messages [ 1104.661320] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:51:20 (1787842280) [ 1119.208779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.217078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.231501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.250432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.256183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.265390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.291031] Lustre: Skipped 1 previous similar message [ 1126.371899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.380432] Lustre: Skipped 1 previous similar message [ 1131.492217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.505909] Lustre: Skipped 1 previous similar message [ 1133.024658] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1133.180454] Lustre: server umount lustre-OST0000 complete [ 1140.195435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1140.203531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.225388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.168562] Lustre: server umount lustre-MDT0000 complete [ 1146.849814] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.873675] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1146.987305] LustreError: 38579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842324 with bad export cookie 17673622562338425948 [ 1146.990910] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.001655] LustreError: 38579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1147.396036] Lustre: server umount lustre-MDT0001 complete [ 1158.398314] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1161.179850] Key type lgssc unregistered [ 1161.508601] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.523943] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.604436] LNet: Removed LNI 192.168.206.118@tcp [ 1163.636715] Key type .llcrypt unregistered [ 1163.644336] Key type ._llcrypt unregistered [ 1181.113557] Key type ._llcrypt registered [ 1181.115816] Key type .llcrypt registered [ 1181.197813] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 1195.501686] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1196.557706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.643654] alg: No test for adler32 (adler32-zlib) [ 1197.723258] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1197.987772] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1199.665945] Key type lgssc registered [ 1200.755893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.760362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.777186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.994966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1236.039086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1236.115752] Lustre: lustre-MDT0000: new disk, initializing [ 1236.171774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.184792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1240.250517] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1249.929684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.145447] Lustre: 46035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1250.223311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1250.230862] Lustre: Skipped 1 previous similar message [ 1250.331389] Lustre: lustre-MDT0001: new disk, initializing [ 1250.464724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.507424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1250.514935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1255.183639] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1266.138347] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.574245] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1279.621505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.049481] Lustre: lustre-OST0000: new disk, initializing [ 1280.058697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1280.070423] Lustre: 48003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1280.186583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.641624] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1287.769047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1287.777014] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1287.888331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1296.841187] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1302.535214] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1302.740953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1308.566481] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1308.860814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1313.763314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.770803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.773309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.777525] Lustre: Skipped 1 previous similar message [ 1317.811332] Lustre: server umount lustre-OST0000 complete [ 1322.470791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.484160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.506924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.509620] Lustre: Skipped 1 previous similar message [ 1324.000768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.006846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.012299] Lustre: Skipped 1 previous similar message [ 1328.631263] Lustre: server umount lustre-MDT0000 complete [ 1333.067680] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842511 with bad export cookie 16661445488322666982 [ 1333.074337] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.087533] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1333.523719] Lustre: server umount lustre-MDT0001 complete [ 1344.154266] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:55:19 (1787842519) [ 1350.372784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.779595] LustreError: 50463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.800768] LustreError: 50463:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1350.869439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.594607] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1356.268429] LustreError: 50464:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.357643] LustreError: 50463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.038794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.296301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.906035] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1376.835330] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1382.385475] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.912780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.117266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.655737] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1408.904388] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1413.712885] LustreError: 52993:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1414.568375] Lustre: Modifying parameter general.timeout=1 in log params [ 1415.598639] Lustre: Modifying parameter general.timeout=2 in log params [ 1418.260517] Lustre: Modifying parameter general.timeout=5 in log params [ 1418.263690] Lustre: Skipped 2 previous similar messages [ 1422.804404] Lustre: Modifying parameter general.timeout=10 in log params [ 1422.808879] Lustre: Skipped 4 previous similar messages [ 1430.869277] Lustre: Modifying parameter general.timeout=19 in log params [ 1430.878094] Lustre: Skipped 8 previous similar messages [ 1447.448516] Lustre: Modifying parameter general.timeout=37 in log params [ 1447.452581] Lustre: Skipped 17 previous similar messages [ 1479.725984] Lustre: Modifying parameter general.timeout=72 in log params [ 1479.736914] Lustre: Skipped 34 previous similar messages [ 1546.261098] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:58:41 (1787842721) [ 1547.692595] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1547.708266] Lustre: Skipped 58 previous similar messages [ 1549.816575] Lustre: 59934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1551.858818] Lustre: 60042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1554.070034] Lustre: 60141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1558.689660] Lustre: 60352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1558.700421] Lustre: 60352:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1591.116622] Lustre: 61815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1591.124297] Lustre: 61815:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1591.153851] LustreError: 61815:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1596.338537] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:59:31 (1787842771) [ 1609.706688] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.710167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.741077] Lustre: Skipped 2 previous similar messages [ 1609.754097] Lustre: Skipped 2 previous similar messages [ 1614.817760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.826740] Lustre: Skipped 2 previous similar messages [ 1615.612898] Lustre: server umount lustre-MDT0000 complete [ 1619.361718] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842797 with bad export cookie 16661445488322667871 [ 1619.364488] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.372715] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1619.655309] Lustre: server umount lustre-MDT0001 complete [ 1634.524549] Lustre: server umount lustre-OST0000 complete [ 1664.127733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.637843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1665.001490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.458560] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1679.131900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.214888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1679.240463] Lustre: 64352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1684.798770] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1693.814320] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1700.417829] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1708.273404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.594605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1708.837468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.844363] Lustre: Skipped 1 previous similar message [ 1715.287850] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1722.964685] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1724.605400] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1724.614140] Lustre: Skipped 22 previous similar messages [ 1724.709886] Lustre: 66928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1732.582443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.594180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.611187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.217363] LustreError: 66208:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.458352] Lustre: server umount lustre-OST0000 complete [ 1743.329789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.358814] Lustre: Skipped 2 previous similar messages [ 1748.222934] Lustre: server umount lustre-MDT0000 complete [ 1748.457336] LustreError: 64361:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.474314] LustreError: 64361:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1751.380265] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842929 with bad export cookie 16661445488322671014 [ 1751.385102] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.391744] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1751.662457] Lustre: server umount lustre-MDT0001 complete [ 1760.896506] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 1763.100952] Key type lgssc unregistered [ 1763.374276] LNet: 68144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.379474] LNetError: 68144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.400302] LNet: Removed LNI 192.168.206.118@tcp [ 1764.025159] Key type .llcrypt unregistered [ 1764.027457] Key type ._llcrypt unregistered [ 1787.245909] Key type ._llcrypt registered [ 1787.247983] Key type .llcrypt registered [ 1787.361609] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 1788.753868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.811700] alg: No test for adler32 (adler32-zlib) [ 1789.965721] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1790.243458] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 1791.920896] Key type lgssc registered [ 1793.084562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.232539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1801.254143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.748418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.028601] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1812.967435] LustreError: 69562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.988046] LustreError: 69562:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1816.665350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.081650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.985379] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1830.941305] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1837.192439] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1845.012673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.825365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.054015] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1859.510419] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1867.070538] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1876.451541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.457910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.488954] Lustre: Skipped 1 previous similar message [ 1881.570540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.581046] Lustre: Skipped 4 previous similar messages [ 1881.798744] Lustre: server umount lustre-MDT0000 complete [ 1885.222162] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843063 with bad export cookie 8044947553041521861 [ 1885.243693] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1885.245811] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.590514] Lustre: server umount lustre-MDT0001 complete [ 1899.905396] Lustre: server umount lustre-OST0000 complete [ 1929.657386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.880613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1929.927658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1930.198665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.258403] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1943.737371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.900787] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1943.959406] Lustre: 74433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1944.443596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.845832] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1956.665550] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1962.328611] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1968.307406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1968.585046] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1968.919160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.474868] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 1981.802744] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1983.440107] Lustre: 77008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1985.093982] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1993.188583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.192995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.204669] Lustre: Skipped 1 previous similar message [ 1993.207843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.706856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.791512] Lustre: server umount lustre-OST0000 complete [ 2000.869400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.878800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.894883] Lustre: Skipped 1 previous similar message [ 2000.898339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.901311] Lustre: Skipped 1 previous similar message [ 2003.936498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.903604] Lustre: server umount lustre-MDT0000 complete [ 2009.045386] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843187 with bad export cookie 8044947553041522694 [ 2009.051809] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2009.052983] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.377303] Lustre: server umount lustre-MDT0001 complete [ 2020.106916] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2022.751680] Key type lgssc unregistered [ 2023.001885] LNet: 78226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.011928] LNetError: 78226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.028465] LNet: Removed LNI 192.168.206.118@tcp [ 2023.884275] Key type .llcrypt unregistered [ 2023.887800] Key type ._llcrypt unregistered [ 2046.326525] Key type ._llcrypt registered [ 2046.329705] Key type .llcrypt registered [ 2046.406419] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2047.535986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.554920] alg: No test for adler32 (adler32-zlib) [ 2048.607112] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2048.874174] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2050.568393] Key type lgssc registered [ 2051.865021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.021452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2059.034988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.501140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.424259] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2071.012273] LustreError: 79637:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.032695] LustreError: 79637:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2072.860713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.400989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2077.530983] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2083.624775] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2088.301791] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.595057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.146656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.052150] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2106.598752] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2113.367980] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2114.203624] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2116.222225] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2116.235411] Lustre: Skipped 1 previous similar message [ 2118.877376] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2118.882743] Lustre: Skipped 2 previous similar messages [ 2122.923989] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2122.929538] Lustre: Skipped 3 previous similar messages [ 2130.970987] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2130.978511] Lustre: Skipped 8 previous similar messages [ 2147.583242] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2147.591396] Lustre: Skipped 19 previous similar messages [ 2180.008423] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2180.015346] Lustre: Skipped 36 previous similar messages [ 2210.211903] Lustre: 87740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2212.902597] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:09:48 (1787843388) [ 2214.862394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2216.607760] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:09:52 (1787843392) [ 2221.036319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2221.047978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.066184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.588817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.596744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.603115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.732574] Lustre: server umount lustre-OST0000 complete [ 2231.777095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.783979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.815660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.824448] Lustre: Skipped 1 previous similar message [ 2233.824500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.841228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.523628] Lustre: server umount lustre-MDT0000 complete [ 2238.947401] LustreError: 79633:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.971704] LustreError: 79633:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2239.687105] LustreError: 79617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843417 with bad export cookie 17104587268708429503 [ 2239.687503] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.698600] LustreError: 79617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2239.987301] Lustre: server umount lustre-MDT0001 complete [ 2249.627387] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2251.993723] Key type lgssc unregistered [ 2252.232708] LNet: 88994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.239382] LNetError: 88994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.261500] LNet: Removed LNI 192.168.206.118@tcp [ 2252.993565] Key type .llcrypt unregistered [ 2252.996775] Key type ._llcrypt unregistered [ 2256.561908] Key type ._llcrypt registered [ 2256.566373] Key type .llcrypt registered [ 2256.665088] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2260.749487] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules [ 2261.814408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.865547] alg: No test for adler32 (adler32-zlib) [ 2262.874979] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2263.094779] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2264.743180] Key type lgssc registered [ 2265.804572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2266.918385] LustreError: 90196:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2281.809798] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2288.803072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2288.903442] LustreError: 90196:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2288.990890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.539963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.512992] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2300.902458] LustreError: 91220:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.926468] LustreError: 91220:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2303.120744] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:11:18 (1787843478) [ 2306.018378] LustreError: 91221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.591118] LustreError: 91252:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2307.604912] LustreError: 91252:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2307.614061] LustreError: 91252:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2307.949317] Lustre: server umount lustre-MDT0000 complete [ 2321.487475] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 2323.972431] Key type lgssc unregistered [ 2324.242709] LNet: 92748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.257520] LNetError: 92748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.272457] LNet: Removed LNI 192.168.206.118@tcp [ 2325.111401] Key type .llcrypt unregistered [ 2325.127511] Key type ._llcrypt unregistered [ 2342.317222] Key type ._llcrypt registered [ 2342.320158] Key type .llcrypt registered [ 2342.413889] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2343.449782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.546031] alg: No test for adler32 (adler32-zlib) [ 2344.656917] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2344.891659] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 2346.583508] Key type lgssc registered [ 2347.582140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2354.654346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2354.686811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.037694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.331219] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2365.419760] LustreError: 93967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.431462] LustreError: 93967:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2368.106429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.694221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2372.220306] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2379.601331] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.421270] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2391.715120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.190666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.805198] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2404.686608] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2444.685713] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:13:40 (1787843620) [ 2446.021838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2447.340383] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:13:43 (1787843623) [ 2450.400659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.409715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.436501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.940314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.941173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.957027] Lustre: Skipped 1 previous similar message [ 2451.969893] Lustre: Skipped 2 previous similar messages [ 2456.565194] Lustre: server umount lustre-MDT0000 complete [ 2457.057228] LustreError: 93962:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.080767] LustreError: 93962:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2459.859259] LustreError: 93947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843637 with bad export cookie 10124411867270550848 [ 2459.870452] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.127573] Lustre: server umount lustre-MDT0001 complete [ 2473.911413] Lustre: server umount lustre-OST0000 complete [ 2485.346950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.887161] LustreError: 98239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.032325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.318168] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2491.361681] LustreError: 98240:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.377460] LustreError: 98240:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2495.482475] LustreError: 98239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.516529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.423538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.129068] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2512.177220] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2518.007076] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2532.363631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.674571] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2532.680207] LustreError: 98230:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2532.699579] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2532.707114] LustreError: 100323:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2532.712602] LustreError: 100323:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2532.717178] LustreError: 100323:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2532.823073] Lustre: server umount lustre-OST0000 complete [ 2532.825199] LustreError: 100323:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2541.260096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2541.407899] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2541.416808] LustreError: 98229:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2541.473175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2541.492078] LustreError: 100790:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2541.505329] LustreError: 100790:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2541.513625] LustreError: 100790:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2541.604731] Lustre: server umount lustre-OST0000 complete [ 2541.611182] LustreError: 100790:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2551.641422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.957643] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2551.962456] Lustre: Found index 0 for lustre-OST0000, updating log [ 2551.971920] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2552.057240] Lustre: lustre-OST0000: new disk, initializing [ 2552.060100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2552.065220] Lustre: 101325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2552.301961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.094783] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2562.900137] Lustre: Failing over lustre-OST0000 [ 2563.061350] Lustre: server umount lustre-OST0000 complete [ 2569.379125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.390852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.396907] Lustre: Skipped 1 previous similar message [ 2569.406804] Lustre: Skipped 1 previous similar message [ 2574.816895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.821745] Lustre: Skipped 1 previous similar message [ 2577.969561] Lustre: server umount lustre-MDT0000 complete [ 2579.940965] LustreError: 98234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.966612] LustreError: 98234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2581.845253] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843759 with bad export cookie 10124411867270553592 [ 2581.857917] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.865075] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2582.219756] Lustre: server umount lustre-MDT0001 complete [ 2591.881367] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:16:07 (1787843767) [ 2610.210297] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2619.642888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.030447] LustreError: 103678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.128344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.536595] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2633.213101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.906325] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2640.957566] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2648.543104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.233749] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2655.708914] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2658.473289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2658.483995] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2658.501541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2663.919634] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2663.937761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2664.233567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.354556] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2664.444851] Lustre: lustre-OST0001: new disk, initializing [ 2664.450592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2664.462657] Lustre: 105998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2664.731856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2664.738623] Lustre: Skipped 2 previous similar messages [ 2665.787572] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2665.808161] Lustre: Skipped 1 previous similar message [ 2665.819246] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2665.819497] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2665.837256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2671.196072] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2671.203573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2671.221615] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2671.335757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2678.963870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2687.520792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2692.068678] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.073854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2692.080371] Lustre: Skipped 2 previous similar messages [ 2692.089991] Lustre: Skipped 1 previous similar message [ 2697.851168] Lustre: server umount lustre-MDT0001 complete [ 2699.314377] LustreError: 105192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.336736] LustreError: 105192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2707.028896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.145195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2707.155946] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2707.226870] Lustre: 103668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2708.426121] LustreError: 107542:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2708.438088] LustreError: 107542:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2708.449377] LustreError: 103670:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2708.525204] LustreError: 107553:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.118@tcp: cfg command failed: rc = -17 [ 2708.535188] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2708.535188] [ 2712.727445] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2713.084530] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2723.516457] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:18:18 (1787843898) [ 2741.162494] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2760.782902] Lustre: 109615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2783.875642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2787.799557] Lustre: 110749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2876.383849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.393327] LustreError: Skipped 2 previous similar messages [ 2876.401880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.423107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.429484] Lustre: Skipped 4 previous similar messages [ 2879.549911] Lustre: server umount lustre-MDT0000 complete [ 2882.019260] LustreError: 103674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.042504] LustreError: 103674:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2883.091343] LustreError: 106011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844061 with bad export cookie 10124411867270554593 [ 2883.095891] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.101852] LustreError: 106011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2883.407431] Lustre: server umount lustre-MDT0001 complete [ 2897.923108] Lustre: server umount lustre-OST0000 complete [ 2912.225567] Lustre: server umount lustre-OST0001 complete [ 2917.468581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2927.186930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2936.117079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2945.775597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2964.028138] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 2974.812353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.841621] Lustre: lustre-MDT0000: reset Object Index mappings [ 2975.632297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.640777] Lustre: Skipped 1 previous similar message [ 2980.454229] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2988.729899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.754561] Lustre: lustre-MDT0001: reset Object Index mappings [ 2994.230608] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 2996.824526] Lustre: 115851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3002.877134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.046336] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3014.828149] LustreError: 116203:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.846355] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3014.847333] LustreError: 116203:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3017.179817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.792162] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3020.792526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3020.817608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3022.951283] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3031.023178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3034.592404] Lustre: 117726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3083.396408] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:24:18 (1787844258) [ 3112.419134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.431319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.448380] Lustre: Skipped 3 previous similar messages [ 3112.460634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.468926] Lustre: Skipped 3 previous similar messages [ 3116.449334] Lustre: server umount lustre-MDT0000 complete [ 3119.998427] LustreError: 115454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844297 with bad export cookie 10124411867270601703 [ 3120.002780] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.007520] LustreError: 115454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3126.538628] Lustre: server umount lustre-MDT0001 complete [ 3129.607714] LustreError: 118480:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x3d:0x0].0x0 (ffff9d16c332be00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3135.811870] Lustre: server umount lustre-OST0000 complete [ 3145.695642] Lustre: server umount lustre-OST0001 complete [ 3152.142571] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 3160.707365] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3195.006790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.209439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3195.245426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3195.306449] Lustre: lustre-MDT0000: new disk, initializing [ 3195.377709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.383347] Lustre: Skipped 3 previous similar messages [ 3195.394770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3198.613376] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3204.552295] Lustre: server umount lustre-MDT0000 complete [ 3212.333377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.585385] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3212.609510] Lustre: 121992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3212.637391] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3217.386474] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3221.754969] Lustre: server umount lustre-MDT0000 complete [ 3229.528418] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:26:45 (1787844405) [ 3230.981540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3232.640766] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:26:48 (1787844408) [ 3248.129154] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 3257.125443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.707651] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3269.170797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.276788] Lustre: 124042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3269.334934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3269.339111] Lustre: Skipped 1 previous similar message [ 3269.395445] Lustre: lustre-MDT0001: new disk, initializing [ 3269.504990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3269.514563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3273.592390] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3277.919618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3282.948129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3283.149570] Lustre: lustre-OST0000: new disk, initializing [ 3283.158413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3283.166994] Lustre: 125620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3284.600681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3284.621434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3284.748784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3288.673383] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3297.846862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.032621] Lustre: lustre-OST0001: new disk, initializing [ 3298.035726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3298.040532] Lustre: 126491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3299.714901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3299.726187] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3299.823996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3303.661068] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 3311.995311] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3314.742823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3670.479553] Lustre: 132908:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3670.502870] Lustre: 132908:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3670.519934] Lustre: 132908:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3670.530059] Lustre: 132908:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3670.541593] Lustre: 132908:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3670.552271] Lustre: 132908:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3967.887901] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:39:02 (1787845142) [ 3985.899839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.898645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.915977] Lustre: Skipped 7 previous similar messages [ 3985.944062] Lustre: Skipped 7 previous similar messages [ 3989.032629] Lustre: server umount lustre-MDT0000 complete [ 3991.026955] LustreError: 124879:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.058341] LustreError: 124879:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 3993.354159] LustreError: 124034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845171 with bad export cookie 10124411867270669960 [ 3993.361342] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.362763] LustreError: 124034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3996.135687] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.142053] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3996.151668] Lustre: Skipped 2 previous similar messages [ 3996.158994] Lustre: Skipped 4 previous similar messages [ 3999.908435] Lustre: server umount lustre-MDT0001 complete [ 4009.894225] Lustre: server umount lustre-OST0000 complete [ 4019.880872] Lustre: server umount lustre-OST0001 complete [ 4026.942453] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 4036.736284] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 4069.991219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.128394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4070.145871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4070.199197] Lustre: lustre-MDT0000: new disk, initializing [ 4070.327486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.333693] Lustre: Skipped 5 previous similar messages [ 4070.353698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4075.904609] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4086.176241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.300400] Lustre: 135961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4086.363956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4086.370293] Lustre: Skipped 1 previous similar message [ 4086.495440] Lustre: lustre-MDT0001: new disk, initializing [ 4086.676318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4086.691991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4090.072800] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4098.729996] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4104.992019] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4112.649230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.132503] Lustre: lustre-OST0000: new disk, initializing [ 4113.138413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4113.148120] Lustre: 137931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4113.291465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.297176] Lustre: Skipped 1 previous similar message [ 4114.571105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4114.579697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4114.721376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4119.450346] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 4130.221481] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.079906] Lustre: lustre-MDD0000: changelog on [ 4143.840816] Lustre: lustre-MDD0001: changelog on [ 6140.466709] Lustre: lustre-MDD0001: changelog off [ 6142.739981] Lustre: 137685:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6144.076573] Lustre: lustre-MDD0000: changelog off [ 6150.627235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.654932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6150.660733] Lustre: Skipped 3 previous similar messages [ 6155.755060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6155.761895] Lustre: Skipped 1 previous similar message [ 6156.593848] Lustre: server umount lustre-OST0000 complete [ 6160.353394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6160.362191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.373920] Lustre: Skipped 1 previous similar message [ 6160.386739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.984042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.993987] Lustre: Skipped 2 previous similar messages [ 6166.632544] Lustre: server umount lustre-MDT0000 complete [ 6169.890308] LustreError: 135953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847347 with bad export cookie 10124411867271487413 [ 6169.895040] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6169.906053] LustreError: 135953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6170.407239] Lustre: server umount lustre-MDT0001 complete [ 6181.472540] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 6184.474458] Key type lgssc unregistered [ 6184.779861] LNet: 147110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.789608] LNetError: 147110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.810250] LNet: Removed LNI 192.168.206.118@tcp [ 6185.710162] Key type .llcrypt unregistered [ 6185.712245] Key type ._llcrypt unregistered [ 6204.344940] Key type ._llcrypt registered [ 6204.347688] Key type .llcrypt registered [ 6204.451837] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6218.978030] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6220.298612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6220.354931] alg: No test for adler32 (adler32-zlib) [ 6221.416555] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6221.620928] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 6223.319236] Key type lgssc registered [ 6224.399388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6256.652556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6256.684830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.961924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6258.005136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6258.111715] Lustre: lustre-MDT0000: new disk, initializing [ 6258.213946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.224721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6263.465673] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6275.255736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.390891] Lustre: 150309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6275.481558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6275.489844] Lustre: Skipped 1 previous similar message [ 6275.643838] Lustre: lustre-MDT0001: new disk, initializing [ 6275.848837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6275.955915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6275.969742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6280.931315] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6292.428328] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6299.277818] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6308.161347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.469505] Lustre: lustre-OST0000: new disk, initializing [ 6308.482766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6308.498596] Lustre: 152279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6308.611851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.591217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6314.612352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6314.733446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6315.064606] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6325.347617] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.298233] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_module obdecho/obdecho [ 6335.586310] LustreError: 153283:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6335.602087] LustreError: 153283:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6337.834613] LustreError: 153380:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6352.866539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6352.877784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.899899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6355.939430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6355.940153] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.957313] Lustre: Skipped 1 previous similar message [ 6357.528734] Lustre: server umount lustre-MDT0000 complete [ 6362.272634] LustreError: 150304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847540 with bad export cookie 14455339191784258599 [ 6362.292304] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6362.764137] Lustre: server umount lustre-MDT0001 complete [ 6377.585303] Lustre: server umount lustre-OST0000 complete [ 6398.463246] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6409.311385] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6448.840794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.092808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6449.124946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6449.188549] Lustre: lustre-MDT0000: new disk, initializing [ 6449.262050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.275908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6454.784969] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6466.267696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.347729] Lustre: 157099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6466.401178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6466.404736] Lustre: Skipped 1 previous similar message [ 6466.467638] Lustre: lustre-MDT0001: new disk, initializing [ 6466.551479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.604824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6466.616041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6471.407428] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6481.110337] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6487.585449] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6495.431027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.693372] Lustre: lustre-OST0000: new disk, initializing [ 6495.698814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6495.709430] Lustre: 159071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6495.792131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.740783] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6505.013951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6505.027967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6505.184117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6511.297433] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6515.873987] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6516.059259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6520.875232] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6521.087179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6525.929570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.935969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.948904] Lustre: Skipped 1 previous similar message [ 6525.949396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6525.966582] Lustre: Skipped 3 previous similar messages [ 6529.218846] Lustre: server umount lustre-OST0000 complete [ 6533.088775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6533.114645] LustreError: Skipped 1 previous similar message [ 6533.122949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.150313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.096519] Lustre: server umount lustre-MDT0000 complete [ 6541.281635] LustreError: 157111:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.299602] LustreError: 157111:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6543.292926] LustreError: 159070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847721 with bad export cookie 14455339191784259558 [ 6543.297834] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.304751] LustreError: 159070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6543.610496] Lustre: server umount lustre-MDT0001 complete [ 6547.017490] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:22:02 (1787847722) [ 6548.772899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6550.839804] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:22:06 (1787847726) [ 6558.587828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.134128] LustreError: 161481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.194672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.449859] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6564.327522] LustreError: 161482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.449694] LustreError: 161481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.990659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.990503] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6585.232989] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.070536] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.004969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.438707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.450361] Lustre: Skipped 1 previous similar message [ 6605.886843] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6613.208285] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6624.337194] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.18@tcp (stopping) [ 6624.343947] Lustre: Skipped 2 previous similar messages [ 6625.186225] Lustre: server umount lustre-MDT0000 complete [ 6628.324473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.333794] LustreError: 161476:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.335939] Lustre: Skipped 3 previous similar messages [ 6628.360856] LustreError: 161476:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6628.706140] LustreError: 161464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847806 with bad export cookie 14455339191784260440 [ 6628.710319] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.713597] LustreError: 161464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6633.447373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6633.455238] Lustre: Skipped 1 previous similar message [ 6638.562521] LustreError: 161477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.578045] LustreError: 161477:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6643.167222] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6643.391628] Lustre: server umount lustre-MDT0001 complete [ 6650.387687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6662.885239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.047673] LustreError: 165326:0:(llog.c:1655:llog_backup()) MGC192.168.206.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6663.056551] Lustre: 165326:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6663.137069] LustreError: 165331:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6663.150749] LustreError: 165331:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6695.979938] LustreError: 165333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.995150] LustreError: 165333:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 24 previous similar messages [ 6723.839410] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6723.845438] Lustre: 165326:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6760.492757] LustreError: 165333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.516071] LustreError: 165333:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 40 previous similar messages [ 6764.549932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6769.636591] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6769.919877] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6779.085579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.209812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6803.972391] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6804.798982] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6814.037688] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6816.015515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6822.582485] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6824.387454] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6844.906225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.907832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.923466] Lustre: Skipped 3 previous similar messages [ 6844.924481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6844.959773] Lustre: Skipped 5 previous similar messages [ 6849.643977] Lustre: server umount lustre-MDT0000 complete [ 6854.201278] LustreError: 164265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848032 with bad export cookie 14455339191784266929 [ 6854.203458] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.209173] LustreError: 164265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6854.783693] Lustre: server umount lustre-MDT0001 complete [ 6868.382699] Lustre: server umount lustre-OST0000 complete [ 6892.647022] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 6905.730484] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 6947.233577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.497521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6947.522898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6947.627154] Lustre: lustre-MDT0000: new disk, initializing [ 6947.724223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.734715] Lustre: Skipped 1 previous similar message [ 6947.754424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6952.310985] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6962.919309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.074301] Lustre: 170436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6963.167188] LustreError: 170448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.189630] LustreError: 170448:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 36 previous similar messages [ 6963.413889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6968.041890] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 6978.226905] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6983.892403] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6992.944515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.519612] Lustre: lustre-OST0000: new disk, initializing [ 6993.531659] Lustre: Skipped 1 previous similar message [ 6993.536902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6993.548146] Lustre: Skipped 2 previous similar messages [ 6993.555808] Lustre: 172406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7002.558042] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7003.197699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7003.211021] Lustre: Skipped 1 previous similar message [ 7003.221218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7003.411187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7013.860749] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7019.500994] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7019.768756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7024.404806] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7024.612219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7029.221968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7029.222933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.234982] LustreError: Skipped 1 previous similar message [ 7029.240639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.256291] Lustre: Skipped 1 previous similar message [ 7029.280867] Lustre: Skipped 1 previous similar message [ 7032.986306] Lustre: server umount lustre-OST0000 complete [ 7043.698182] Lustre: server umount lustre-MDT0000 complete [ 7047.538831] LustreError: 171231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848225 with bad export cookie 14455339191784283736 [ 7047.553094] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.569493] LustreError: 171231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7048.151585] Lustre: server umount lustre-MDT0001 complete [ 7057.924746] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:30:33 (1787848233) [ 7065.185096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.895313] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7080.479107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.821846] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7094.169044] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7100.865594] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7108.035313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.169451] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7123.060202] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7137.184749] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:31:52 (1787848312) [ 7146.467094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.476563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7146.478030] Lustre: Skipped 3 previous similar messages [ 7146.478870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7146.478875] Lustre: Skipped 2 previous similar messages [ 7146.493085] LustreError: Skipped 1 previous similar message [ 7148.082440] Lustre: server umount lustre-OST0000 complete [ 7157.999832] Lustre: server umount lustre-MDT0000 complete [ 7161.475373] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848339 with bad export cookie 14455339191784284618 [ 7161.479377] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7161.484899] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7161.717394] Lustre: server umount lustre-MDT0001 complete [ 7173.866646] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7176.808484] Key type lgssc unregistered [ 7177.199602] LNet: 178550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7177.209912] LNetError: 178550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7177.228956] LNet: Removed LNI 192.168.206.118@tcp [ 7178.086532] Key type .llcrypt unregistered [ 7178.089762] Key type ._llcrypt unregistered [ 7196.883936] Key type ._llcrypt registered [ 7196.887471] Key type .llcrypt registered [ 7197.005520] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7198.425908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7198.451063] alg: No test for adler32 (adler32-zlib) [ 7199.545140] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7199.838948] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7201.559726] Key type lgssc registered [ 7202.516494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7210.409696] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7210.425814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.079377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7231.199328] LustreError: 179792:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.118@tcp: failed processing log, type 4: rc = -110 [ 7236.319731] LustreError: 179792:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7236.337900] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7236.352828] LustreError: MGC192.168.206.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7236.366411] LustreError: 179792:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7236.374419] LustreError: 179792:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7236.379729] LustreError: 179792:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7236.467419] Lustre: server umount lustre-OST0000 complete [ 7236.472883] LustreError: 179792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7245.042383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.536679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.891983] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7256.034373] LustreError: 180292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.060715] LustreError: 180292:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7259.523445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.907969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7264.511422] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7272.636895] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7278.523551] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7286.835750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7287.056945] LustreError: 182108:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7287.064459] Lustre: 182108:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7287.184961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.198155] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7300.445843] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7304.548501] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.562973] Lustre: Skipped 2 previous similar messages [ 7304.563985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7304.576886] Lustre: Skipped 2 previous similar messages [ 7308.777168] Lustre: server umount lustre-MDT0000 complete [ 7309.812664] LustreError: 180292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.845915] LustreError: 180292:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7312.730430] LustreError: 180272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848490 with bad export cookie 14382165991731060029 [ 7312.745917] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.751175] LustreError: 180272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7312.799263] LustreError: 181022:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dd844717 x1874694899790208/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7313.106540] Lustre: server umount lustre-MDT0001 complete [ 7327.930306] Lustre: server umount lustre-OST0000 complete [ 7337.251562] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7347.216201] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7383.037321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.321455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7383.375766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7383.490150] Lustre: lustre-MDT0000: new disk, initializing [ 7383.553799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.578542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7387.388788] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7396.682746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.766647] Lustre: 185634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7396.831810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7396.839279] Lustre: Skipped 1 previous similar message [ 7396.937479] Lustre: lustre-MDT0001: new disk, initializing [ 7397.031261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.070322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7397.086098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7401.507825] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7410.671439] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7416.612960] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7423.193989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.583987] Lustre: lustre-OST0000: new disk, initializing [ 7423.593816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7423.599724] Lustre: 187604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7423.693942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.935926] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7431.235056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7431.250836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7431.302723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7439.156508] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7444.848192] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7445.082849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7449.867899] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7450.076237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7452.127951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7452.130454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7452.138552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7452.159773] Lustre: Skipped 1 previous similar message [ 7457.256661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.270858] Lustre: Skipped 2 previous similar messages [ 7458.361623] Lustre: server umount lustre-OST0000 complete [ 7462.368857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.377373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7462.397471] Lustre: Skipped 1 previous similar message [ 7462.414954] Lustre: Skipped 1 previous similar message [ 7467.492430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7467.505844] Lustre: Skipped 1 previous similar message [ 7468.378465] Lustre: server umount lustre-MDT0000 complete [ 7472.301748] LustreError: 186439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848650 with bad export cookie 14382165991731060757 [ 7472.306181] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.318196] LustreError: 186439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7472.748549] Lustre: server umount lustre-MDT0001 complete [ 7481.579706] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:37:36 (1787848656) [ 7499.739235] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 7502.718510] Key type lgssc unregistered [ 7503.059838] LNet: 190551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.071224] LNetError: 190551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.091373] LNet: Removed LNI 192.168.206.118@tcp [ 7503.958491] Key type .llcrypt unregistered [ 7503.966086] Key type ._llcrypt unregistered [ 7521.370233] Key type ._llcrypt registered [ 7521.373710] Key type .llcrypt registered [ 7521.489580] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7522.650405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7522.690469] alg: No test for adler32 (adler32-zlib) [ 7523.833950] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7524.027689] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 7525.695085] Key type lgssc registered [ 7526.898568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7535.215172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7535.240217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.968601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.172078] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7546.349627] LustreError: 191769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.368633] LustreError: 191769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7550.404616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.793087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7555.022349] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7563.253591] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7570.097478] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7578.549040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.758618] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7578.762415] LustreError: 193634:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7578.773538] Lustre: 193634:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7578.896262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7585.526498] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7593.954785] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7597.024272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7597.035649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.055550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.608465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.650960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.662796] Lustre: Skipped 2 previous similar messages [ 7603.245869] Lustre: server umount lustre-MDT0000 complete [ 7605.759807] LustreError: 191770:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.787899] LustreError: 191770:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7608.026906] LustreError: 191749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848786 with bad export cookie 318564619237813299 [ 7608.045438] LustreError: 191749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7608.050249] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7608.398203] Lustre: server umount lustre-MDT0001 complete [ 7623.271447] Lustre: server umount lustre-OST0000 complete [ 7633.721831] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 7643.207310] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7682.217182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.638778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7682.661964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7682.722610] Lustre: lustre-MDT0000: new disk, initializing [ 7682.837338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.875655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7689.334162] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7700.481912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.617310] Lustre: 197163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7700.662178] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7700.665313] Lustre: Skipped 1 previous similar message [ 7700.669415] LustreError: 197946:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7700.676382] LustreError: 197946:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7700.683802] Lustre: 197946:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7700.711183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7700.716698] Lustre: Skipped 1 previous similar message [ 7700.805881] Lustre: lustre-MDT0001: new disk, initializing [ 7700.913701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7700.983581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7701.000948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7706.423301] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7716.932838] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7723.514605] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7732.307187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.674029] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7732.678258] Lustre: Skipped 1 previous similar message [ 7732.684762] LustreError: 199109:0:(llog.c:1696:llog_backup()) MGC192.168.206.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7732.694620] LustreError: 199109:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7732.703988] Lustre: 199109:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7732.716389] Lustre: 199109:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7732.743620] Lustre: lustre-OST0000: new disk, initializing [ 7732.759946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7732.776987] Lustre: 199132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7732.898340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.924291] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7741.002975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7741.022508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7741.129360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7748.643204] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7753.322096] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7753.481885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7757.724954] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7757.963436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7761.888382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7761.892924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.915119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.944956] Lustre: Skipped 2 previous similar messages [ 7766.209745] Lustre: server umount lustre-OST0000 complete [ 7772.136932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7772.143155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.163277] Lustre: Skipped 1 previous similar message [ 7772.178646] Lustre: Skipped 2 previous similar messages [ 7776.605834] Lustre: server umount lustre-MDT0000 complete [ 7777.250053] LustreError: 199428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.267931] LustreError: 199428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7780.044335] LustreError: 197155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848958 with bad export cookie 318564619237814020 [ 7780.050300] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7780.056464] LustreError: 197155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7780.506299] Lustre: server umount lustre-MDT0001 complete [ 7789.367350] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:42:44 (1787848964) [ 7806.990660] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 7817.277288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.572239] LustreError: 202258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7817.646088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.988093] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7822.833815] LustreError: 202259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.942621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.760606] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7837.978260] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7845.399316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.571982] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7861.003827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7861.337704] Lustre: lustre-OST0001: new disk, initializing [ 7861.344805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7861.355690] Lustre: 204565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7861.471858] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7861.482341] Lustre: Skipped 2 previous similar messages [ 7868.180955] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7871.592660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7871.598717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7871.739858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7878.643486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7884.902378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7886.900500] Lustre: DEBUG MARKER: ADD OST3 [ 7891.581806] loop: module loaded [ 7891.675322] loop0: detected capacity change from 0 to 204800000 [ 7891.904220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7891.913366] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7891.948276] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7897.781347] loop0: detected capacity change from 0 to 204800000 [ 7899.127560] Lustre: DEBUG MARKER: STOP OST3 [ 7904.349649] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.474036] Lustre: lustre-OST0003: new disk, initializing [ 7904.478716] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7904.489476] Lustre: 206551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7906.148637] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7906.298341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7906.305144] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7906.349534] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7908.036494] Lustre: Failing over lustre-OST0003 [ 7908.191179] LustreError: 206739:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7908.228080] Lustre: server umount lustre-OST0003 complete [ 7910.793384] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7911.409691] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.438643] LustreError: 204273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.456658] LustreError: 204273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7919.819642] Lustre: DEBUG MARKER: START OST3 again [ 7927.470627] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.823088] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7927.835923] Lustre: Skipped 1 previous similar message [ 7927.864633] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7928.870695] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7929.784990] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7929.786143] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7934.673563] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 7956.484250] Lustre: Failing over lustre-OST0003 [ 7956.790306] Lustre: server umount lustre-OST0003 complete [ 7958.514949] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.517240] LustreError: 203746:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.526304] Lustre: Skipped 2 previous similar messages [ 7958.553604] LustreError: 203746:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 7978.982992] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.986694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7979.009520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7979.011988] Lustre: Skipped 2 previous similar messages [ 7979.021427] LustreError: Skipped 1 previous similar message [ 7980.454221] Lustre: server umount lustre-MDT0000 complete [ 7983.694220] LustreError: 202240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849161 with bad export cookie 318564619237814895 [ 7983.695829] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.705657] LustreError: 202240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7988.128857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7988.132730] Lustre: Skipped 5 previous similar messages [ 7990.175344] Lustre: server umount lustre-MDT0001 complete [ 7994.742531] Lustre: server umount lustre-OST0000 complete [ 7998.536867] Lustre: server umount lustre-OST0001 complete [ 8005.829101] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8016.612035] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8055.827453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.071414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8056.108692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8056.197992] Lustre: lustre-MDT0000: new disk, initializing [ 8056.290061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.353584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8060.646427] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8070.971272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.074520] Lustre: 212028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8071.340121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8076.239672] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8087.386601] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8096.081375] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8104.566623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8104.838105] Lustre: lustre-OST0000: new disk, initializing [ 8104.842063] Lustre: Skipped 1 previous similar message [ 8104.848143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8104.851451] Lustre: Skipped 2 previous similar messages [ 8104.856742] Lustre: 213997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8106.775694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8106.785219] Lustre: Skipped 1 previous similar message [ 8106.797162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8106.902340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.065859] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8120.562196] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8125.988147] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8126.206713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8130.334346] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8130.521663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8132.584768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8132.593663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.605033] Lustre: Skipped 3 previous similar messages [ 8132.609730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.741374] Lustre: server umount lustre-OST0000 complete [ 8142.817704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8142.821733] LustreError: Skipped 1 previous similar message [ 8148.713829] Lustre: server umount lustre-MDT0000 complete [ 8152.399734] LustreError: 212021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849330 with bad export cookie 318564619237817198 [ 8152.408852] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8152.412753] LustreError: 212021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8152.858243] Lustre: server umount lustre-MDT0001 complete [ 8156.153801] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:48:51 (1787849331) [ 8171.552245] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8182.464966] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8221.339683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.653208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8221.694583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8221.810203] Lustre: lustre-MDT0000: new disk, initializing [ 8221.932106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8226.390484] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8236.135792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.291720] Lustre: 218606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8236.556905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8241.005321] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8249.684621] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8255.778703] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8263.045674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.314896] Lustre: 220577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.230543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8269.993506] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8279.752784] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8284.988777] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8285.190762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8290.845546] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8291.156392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8295.904152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.908273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.912857] LustreError: Skipped 1 previous similar message [ 8295.924295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8295.930348] Lustre: Skipped 3 previous similar messages [ 8295.959220] Lustre: Skipped 8 previous similar messages [ 8308.191229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8308.682466] Lustre: server umount lustre-OST0000 complete [ 8319.849809] Lustre: server umount lustre-MDT0000 complete [ 8321.510580] LustreError: 218614:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8321.537964] LustreError: 218614:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 8324.348416] LustreError: 218598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849502 with bad export cookie 318564619237818073 [ 8324.382885] LustreError: 218598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8324.392583] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8325.375903] Lustre: server umount lustre-MDT0001 complete [ 8339.347307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.348987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.354347] Lustre: Skipped 5 previous similar messages [ 8348.334911] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8358.893445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.518792] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8373.914242] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.473177] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8389.577986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8397.270183] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8406.026255] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8424.928195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.942658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.946678] Lustre: Skipped 2 previous similar messages [ 8424.950098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8424.956795] Lustre: Skipped 6 previous similar messages [ 8429.220647] Lustre: server umount lustre-OST0000 complete [ 8439.451708] Lustre: server umount lustre-MDT0000 complete [ 8442.346964] LustreError: 222798:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.364408] LustreError: 222798:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 8443.056985] LustreError: 222779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849621 with bad export cookie 318564619237818962 [ 8443.067704] LustreError: 222779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8443.067893] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8443.308583] Lustre: server umount lustre-MDT0001 complete [ 8544.815284] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 8547.325221] Key type lgssc unregistered [ 8547.630559] LNet: 226486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8547.635224] LNetError: 226486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8547.647859] LNet: Removed LNI 192.168.206.118@tcp [ 8548.562144] Key type .llcrypt unregistered [ 8548.566934] Key type ._llcrypt unregistered [ 8575.150593] Key type ._llcrypt registered [ 8575.155969] Key type .llcrypt registered [ 8575.272890] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8592.170760] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8593.558376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.629496] alg: No test for adler32 (adler32-zlib) [ 8594.675582] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 8594.908706] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 8596.543169] Key type lgssc registered [ 8597.635986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8635.042129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.080175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8636.278590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8636.302337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8636.374467] Lustre: lustre-MDT0000: new disk, initializing [ 8636.425087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.451460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8640.985466] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8651.112243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.278677] Lustre: 229844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8651.329768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8651.334371] Lustre: Skipped 1 previous similar message [ 8651.472695] Lustre: lustre-MDT0001: new disk, initializing [ 8651.619307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8651.665394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8651.673900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8655.991437] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8664.930746] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8670.296741] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8676.822438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.037914] Lustre: lustre-OST0000: new disk, initializing [ 8677.047434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8677.056386] Lustre: 231810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8677.118696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.219336] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8685.101347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8685.110182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8685.206310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8689.648258] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8693.666462] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8693.964867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8698.295000] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8698.509755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8700.898258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8700.902557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.907396] LustreError: Skipped 1 previous similar message [ 8700.912346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8700.942054] Lustre: Skipped 1 previous similar message [ 8706.021546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8706.028208] Lustre: Skipped 2 previous similar messages [ 8706.464829] Lustre: server umount lustre-OST0000 complete [ 8711.139179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.143286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8711.156329] Lustre: Skipped 1 previous similar message [ 8716.152922] Lustre: server umount lustre-MDT0000 complete [ 8719.238706] LustreError: 229836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849897 with bad export cookie 8243048095049721108 [ 8719.246986] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8719.253399] LustreError: 229836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8719.553468] Lustre: server umount lustre-MDT0001 complete [ 8736.977995] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8746.536772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.938024] LustreError: 234743:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8746.960453] LustreError: 234743:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8746.989231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.968937] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8752.096823] LustreError: 234744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.197766] LustreError: 234743:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.199520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.312183] LustreError: 234744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.673140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8766.091287] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8768.867879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8775.204030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8775.606224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.658987] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8789.894895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8790.072376] Lustre: lustre-OST0001: new disk, initializing [ 8790.075469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8790.081271] Lustre: 237050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8796.504479] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8799.380510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8799.394966] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8799.522484] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8807.449598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8814.662893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8830.285266] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:00:05 (1787850005) [ 8833.503919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8833.524096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.554552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.560382] Lustre: Skipped 1 previous similar message [ 8835.559262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.576543] Lustre: Skipped 2 previous similar messages [ 8840.673395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.685545] Lustre: Skipped 5 previous similar messages [ 8847.327164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8847.529361] Lustre: server umount lustre-MDT0000 complete [ 8850.692646] LustreError: 234723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850028 with bad export cookie 8243048095049721990 [ 8850.696441] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8850.699465] LustreError: 234723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8850.914116] LustreError: 238184:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.915915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.924075] LustreError: 238184:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8850.940207] Lustre: Skipped 1 previous similar message [ 8851.016465] Lustre: server umount lustre-MDT0001 complete [ 8856.554663] Lustre: server umount lustre-OST0000 complete [ 8859.885451] Lustre: server umount lustre-OST0001 complete [ 8865.879534] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 8875.343925] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 8912.885282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.115856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8913.135626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8913.218419] Lustre: lustre-MDT0000: new disk, initializing [ 8913.290687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8913.295215] Lustre: Skipped 1 previous similar message [ 8913.310767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8917.504833] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8927.773050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.917939] Lustre: 241008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8927.996950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8928.000299] Lustre: Skipped 1 previous similar message [ 8928.116210] Lustre: lustre-MDT0001: new disk, initializing [ 8928.258971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8928.274299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8932.141642] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8941.304672] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8947.347720] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8954.597057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8955.108872] Lustre: lustre-OST0000: new disk, initializing [ 8955.113203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8955.133299] Lustre: 242979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8955.274985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.281701] Lustre: Skipped 1 previous similar message [ 8957.106701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8957.121680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8957.310604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8962.396882] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 8972.848522] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8978.213150] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8978.432107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8982.611619] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8982.907822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8988.129816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8988.132173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.134149] LustreError: Skipped 1 previous similar message [ 8988.147516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8988.158904] Lustre: Skipped 1 previous similar message [ 8988.196521] Lustre: Skipped 5 previous similar messages [ 8999.394992] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8999.802233] Lustre: server umount lustre-OST0000 complete [ 9005.027364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9005.045680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.059235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.063159] Lustre: Skipped 4 previous similar messages [ 9010.905081] Lustre: server umount lustre-MDT0000 complete [ 9013.729621] LustreError: 241019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9013.751114] LustreError: 241019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9014.638067] LustreError: 240999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850192 with bad export cookie 8243048095049724125 [ 9014.643146] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9014.655864] LustreError: 240999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9015.059745] Lustre: server umount lustre-MDT0001 complete [ 9023.604917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.169571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.479644] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9034.723867] LustreError: 245195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.735889] LustreError: 245195:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9037.609438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.600318] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9052.406380] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9058.346720] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9065.451209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9072.232581] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9079.513862] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9223.649979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9223.666459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.685980] Lustre: Skipped 1 previous similar message [ 9223.696895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9223.711140] Lustre: Skipped 2 previous similar messages [ 9229.617445] Lustre: server umount lustre-OST0000 complete [ 9239.398807] Lustre: server umount lustre-MDT0000 complete [ 9241.057275] LustreError: 245195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9241.068929] LustreError: 245195:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9242.658719] LustreError: 245177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850420 with bad export cookie 8243048095049725007 [ 9242.664745] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9242.667734] LustreError: 245177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9242.919878] Lustre: server umount lustre-MDT0001 complete [ 9348.296215] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [ 9350.866338] Key type lgssc unregistered [ 9351.207482] LNet: 248805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9351.216067] LNetError: 248805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9351.229750] LNet: Removed LNI 192.168.206.118@tcp [ 9352.054159] Key type .llcrypt unregistered [ 9352.057638] Key type ._llcrypt unregistered [ 9378.887308] Key type ._llcrypt registered [ 9378.890187] Key type .llcrypt registered [ 9379.042144] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 9395.766275] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 9397.364393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9397.388568] alg: No test for adler32 (adler32-zlib) [ 9398.460803] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 9398.763901] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [ 9400.479159] Key type lgssc registered [ 9401.470181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9444.417193] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 9454.552208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9454.574230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.843807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9455.946783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9456.046566] Lustre: lustre-MDT0000: new disk, initializing [ 9456.161616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.179819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9460.452708] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9471.617216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.739723] Lustre: 252882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9471.788955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9471.794518] Lustre: Skipped 1 previous similar message [ 9471.895923] Lustre: lustre-MDT0001: new disk, initializing [ 9472.017385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9472.055961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9472.071617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9477.044113] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9481.713563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9487.638319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.942345] Lustre: lustre-OST0000: new disk, initializing [ 9487.951270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9487.962263] Lustre: 254512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9488.062936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.112535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9490.163475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9490.374267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9494.759646] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9505.639194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9505.883446] Lustre: lustre-OST0001: new disk, initializing [ 9505.887735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9505.901377] Lustre: 255386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9506.001208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9512.743606] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9513.546843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9513.578474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9513.677337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9524.582312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9530.490414] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9543.653198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9543.666536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.681408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9544.687697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.691702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9544.721151] Lustre: Skipped 1 previous similar message [ 9544.737230] Lustre: Skipped 3 previous similar messages [ 9548.828328] Lustre: server umount lustre-MDT0000 complete [ 9553.087523] LustreError: 252873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850731 with bad export cookie 12512459642645708502 [ 9553.103598] LustreError: 252873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9553.112809] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.702120] Lustre: server umount lustre-MDT0001 complete [ 9569.017813] Lustre: server umount lustre-OST0000 complete [ 9570.271814] Lustre: 250387:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850732/real 1787850732] req@ffff9d15c15a7c00 x1874697205657344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9570.301789] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.310096] Lustre: Skipped 1 previous similar message [ 9573.015988] Lustre: server umount lustre-OST0001 complete [ 9585.768656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9595.762658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9604.855745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9617.003891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9636.653337] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 9646.787356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.814864] Lustre: lustre-MDT0000: reset Object Index mappings [ 9647.170240] LustreError: 260393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.183597] LustreError: 260393:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9647.237167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9652.686491] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9652.708398] LustreError: 260394:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.825489] LustreError: 260393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.945717] LustreError: 260394:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.889757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9667.933920] Lustre: lustre-MDT0001: reset Object Index mappings [ 9668.071092] LustreError: 260393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.543315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9677.161063] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9680.543389] Lustre: 261536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9687.621366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9687.963027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.082210] LustreError: 261891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.112727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9695.375877] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9704.495801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9710.072751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9712.450924] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9720.617366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9724.378160] Lustre: 263409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9730.016768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9730.030793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.046473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9735.124172] Lustre: server umount lustre-MDT0000 complete [ 9735.664535] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9735.689149] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 9738.777379] LustreError: 260375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850916 with bad export cookie 12512459642645715292 [ 9738.780741] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9738.797984] LustreError: 260375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9739.250621] Lustre: server umount lustre-MDT0001 complete [ 9753.594906] Lustre: server umount lustre-OST0000 complete [ 9756.127258] Lustre: 250386:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850918/real 1787850918] req@ffff9d15c2284f00 x1874697205710208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9756.167695] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.194825] Lustre: Skipped 3 previous similar messages [ 9757.169644] Lustre: server umount lustre-OST0001 complete [ 9775.094581] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:15:50 (1787850950) [ 9790.846113] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [ 9800.772686] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 9831.315726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.489924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9831.516932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9831.624274] Lustre: lustre-MDT0000: new disk, initializing [ 9831.687425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.692553] Lustre: Skipped 1 previous similar message [ 9831.702453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9835.409280] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9844.101543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.195350] Lustre: 267659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9844.246566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9844.252619] Lustre: Skipped 1 previous similar message [ 9844.338254] Lustre: lustre-MDT0001: new disk, initializing [ 9844.456663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9844.469682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9848.549302] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9857.300021] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9863.179226] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9870.196771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9870.408212] Lustre: lustre-OST0000: new disk, initializing [ 9870.411697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9870.416266] Lustre: 269628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9871.808750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9871.827895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9871.989440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9876.978693] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9886.701501] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9890.863706] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9891.035387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9895.298864] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9895.473743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9897.442700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9897.446117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.452660] LustreError: Skipped 1 previous similar message [ 9897.453469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9897.453473] Lustre: Skipped 3 previous similar messages [ 9897.479769] Lustre: Skipped 1 previous similar message [ 9902.560937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9902.572321] Lustre: Skipped 2 previous similar messages [ 9911.265801] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9911.401700] Lustre: server umount lustre-OST0000 complete [ 9916.383853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9916.391388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.410964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.417184] Lustre: Skipped 2 previous similar messages [ 9921.560607] Lustre: server umount lustre-MDT0000 complete [ 9923.044788] LustreError: 267672:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.070423] LustreError: 267672:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9925.455580] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851103 with bad export cookie 12512459642645717378 [ 9925.460996] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9925.471351] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9925.785826] Lustre: server umount lustre-MDT0001 complete [ 9943.357177] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [ 9952.550967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.973580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.979995] Lustre: Skipped 2 previous similar messages [ 9957.432699] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9965.400032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.637070] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9972.291454] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9978.254378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.264837] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [ 9990.487279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9990.627446] Lustre: lustre-OST0001: new disk, initializing [ 9990.630448] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9990.635699] Lustre: 274869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9992.036149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9992.050804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9992.132725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9995.687155] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10003.375612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10006.413664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10010.731724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10011.054298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10017.252226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10017.265525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.276822] Lustre: Skipped 1 previous similar message [10017.283198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10017.293379] Lustre: Skipped 2 previous similar messages [10022.412470] Lustre: server umount lustre-MDT0000 complete [10022.883241] LustreError: 272564:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.892682] LustreError: 272564:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10025.286462] LustreError: 272544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851203 with bad export cookie 12512459642645718267 [10025.290467] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.298700] LustreError: 272544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10025.607436] Lustre: server umount lustre-MDT0001 complete [10039.059860] Lustre: server umount lustre-OST0000 complete [10052.553914] Lustre: server umount lustre-OST0001 complete [10071.737062] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [10084.835934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10085.174342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10085.185436] Lustre: Skipped 3 previous similar messages [10091.189802] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10102.979898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.578420] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10111.633950] Lustre: 279066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10119.235307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.728042] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10130.059856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10137.072320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10142.731682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10144.105675] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10151.980567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10155.932177] Lustre: 280944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10161.409775] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10161.446964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10169.937867] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:22:25 (1787851345) [10185.698815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10185.708871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.728379] Lustre: Skipped 3 previous similar messages [10185.743881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.752880] Lustre: Skipped 3 previous similar messages [10190.583297] Lustre: server umount lustre-MDT0000 complete [10193.847472] LustreError: 280255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851371 with bad export cookie 12512459642645724581 [10193.857586] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10193.861961] LustreError: 280255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10193.895814] LustreError: 277923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10193.908754] LustreError: 277923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [10208.223170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10208.514385] Lustre: server umount lustre-MDT0001 complete [10218.330816] Lustre: server umount lustre-OST0000 complete [10236.383324] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10236.590450] Lustre: server umount lustre-OST0001 complete [10243.067753] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [10252.236235] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [10299.377128] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [10312.023825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.349886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10312.376294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10312.430612] Lustre: lustre-MDT0000: new disk, initializing [10312.556054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10317.539981] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10327.933486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.121213] Lustre: 285188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10328.406788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10332.965927] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10346.990353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10347.420732] Lustre: 286819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10347.636843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10347.647660] Lustre: Skipped 5 previous similar messages [10349.144182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10354.474778] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10365.856750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10366.086423] Lustre: 287694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10367.596226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10367.620524] Lustre: Skipped 1 previous similar message [10367.790525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10373.828715] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10384.448905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10387.847450] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10387.852793] Lustre: Skipped 1 previous similar message [10402.745676] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:26:18 (1787851578) [10405.346528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10405.353182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10405.371514] Lustre: Skipped 5 previous similar messages [10405.377656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10405.383207] Lustre: Skipped 12 previous similar messages [10407.118419] Lustre: server umount lustre-MDT0000 complete [10411.084486] LustreError: 285178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851589 with bad export cookie 12512459642645732442 [10411.086976] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10411.094655] LustreError: 285178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10411.497917] Lustre: server umount lustre-MDT0001 complete [10425.583774] Lustre: server umount lustre-OST0000 complete [10439.763084] Lustre: server umount lustre-OST0001 complete [10447.488538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.368817] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10452.977169] LustreError: 289944:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.000254] LustreError: 289944:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [10461.381255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.603486] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10474.732150] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10481.815923] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10488.953513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.058978] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10508.277622] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10528.881278] loop1: detected capacity change from 0 to 204800000 [10529.213656] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10529.222804] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10529.239641] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10534.380504] loop1: detected capacity change from 0 to 204800000 [10541.481119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.752622] LustreError: 289933:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3320: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10541.791805] LustreError: 289933:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3320 [10541.815590] LustreError: 289933:0:(mgs_handler.c:517:mgs_target_reg()) fs3320-MDT0001: Error checking target index: rc = -13 [10541.854835] LustreError: fs3320-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10541.879366] LustreError: 293660:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10541.897431] LustreError: 293660:0:(tgt_mount.c:2129:server_put_super()) no obd fs3320-MDT0001 [10541.910518] LustreError: 293660:0:(tgt_mount.c:116:server_deregister_mount()) fs3320-MDT0001 not registered [10541.987662] Lustre: server umount fs3320-MDT0001 complete [10541.989679] LustreError: 293660:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10553.747089] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.861837] Lustre: Setting parameter fs3320-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3320-MDT0001 [10554.014449] Lustre: srv-fs3320-MDT0001: No data found on store. Initialize space. [10554.019975] Lustre: Skipped 4 previous similar messages [10554.162062] Lustre: fs3320-MDT0001: new disk, initializing [10554.168461] Lustre: Skipped 3 previous similar messages [10554.415168] Lustre: srv-fs3320-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10554.423911] Lustre: 294128:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3320-MDT0001-mdtlov: can't create new llog: rc = -115 [10554.433687] LustreError: 294128:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10561.803744] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10570.305895] Lustre: server umount fs3320-MDT0001 complete [10573.302042] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10584.143515] loop1: detected capacity change from 0 to 204800000 [10590.205196] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.661481] Lustre: 295695:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3320-MDT0001-mdtlov: can't create new llog: rc = -115 [10590.671377] LustreError: 295695:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10595.292379] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10599.878255] Lustre: server umount fs3320-MDT0001 complete [10610.146475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10615.682916] Lustre: server umount lustre-MDT0000 complete [10624.305391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.387639] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10629.001326] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10629.641651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10637.309234] loop1: detected capacity change from 0 to 204800000 [10644.025176] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.433766] Lustre: 298141:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3320-MDT0001-mdtlov: can't create new llog: rc = -115 [10644.441860] LustreError: 298141:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10648.849936] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10653.486600] Lustre: server umount fs3320-MDT0001 complete [10675.690198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.691400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10675.708171] Lustre: Skipped 7 previous similar messages [10675.719652] LustreError: Skipped 2 previous similar messages [10680.793785] Lustre: server umount lustre-OST0000 complete [10691.974656] Lustre: server umount lustre-MDT0000 complete [10696.117662] LustreError: 289924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851874 with bad export cookie 12512459642645735123 [10696.119622] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.121972] LustreError: 289924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10696.323807] Lustre: server umount lustre-MDT0001 complete [10708.419698] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [10711.145632] Key type lgssc unregistered [10711.476960] LNet: 300491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10711.492790] LNetError: 300491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10711.535672] LNet: Removed LNI 192.168.206.118@tcp [10712.589410] Key type .llcrypt unregistered [10712.598399] Key type ._llcrypt unregistered [10732.636760] Key type ._llcrypt registered [10732.639204] Key type .llcrypt registered [10732.766391] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [10733.971690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10734.060232] alg: No test for adler32 (adler32-zlib) [10735.183989] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [10735.415169] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [10737.055154] Key type lgssc registered [10738.221927] Lustre: Echo OBD driver; http://www.lustre.org/ [10746.671733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10746.689450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.334689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10752.735261] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10758.632894] LustreError: 301764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10758.648270] LustreError: 301764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10761.023428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.768242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10766.897731] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10775.138310] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10780.539887] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10788.036530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10788.651251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10793.722882] LustreError: 303595:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.360394] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10799.074605] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10804.215968] LustreError: 303594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.398587] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10811.370821] LustreError: 303594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.383118] LustreError: 303594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10821.626172] LustreError: 303595:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.655592] LustreError: 303595:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10824.585912] loop1: detected capacity change from 0 to 204800000 [10824.875819] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10824.895331] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10824.923417] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10831.498354] loop1: detected capacity change from 0 to 204800000 [10837.560151] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.755050] Lustre: fs5861-OST0000: new disk, initializing [10837.763267] Lustre: srv-fs5861-OST0000: No data found on store. Initialize space. [10837.781452] Lustre: 305499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.858193] Lustre: fs5861-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.090744] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.114833] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [10845.184842] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10855.287189] Lustre: server umount fs5861-OST0000 complete [10869.579483] loop1: detected capacity change from 0 to 204800000 [10872.813127] LustreError: 303595:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.832507] LustreError: 303595:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [10872.846507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10875.401201] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10875.729269] Lustre: fs5861-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10881.815161] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10888.168945] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10890.836580] loop2: detected capacity change from 0 to 204800000 [10891.074620] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10891.100237] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10891.128206] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10896.138823] loop2: detected capacity change from 0 to 204800000 [10900.846279] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10900.942220] LustreError: fs5861-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10900.962631] LustreError: fs5861-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10900.967828] LustreError: 308539:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10900.973601] LustreError: 308539:0:(tgt_mount.c:2129:server_put_super()) no obd fs5861-OST0001 [10900.977157] LustreError: 308539:0:(tgt_mount.c:116:server_deregister_mount()) fs5861-OST0001 not registered [10901.014227] Lustre: server umount fs5861-OST0001 complete [10901.017087] LustreError: 308539:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10910.520728] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.636588] Lustre: fs5861-OST0001: new disk, initializing [10910.644535] Lustre: srv-fs5861-OST0001: No data found on store. Initialize space. [10910.651477] Lustre: 309091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10910.722572] Lustre: fs5861-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10917.204935] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10926.949871] Lustre: server umount fs5861-OST0000 complete [10938.484325] Lustre: server umount fs5861-OST0001 complete [10939.376236] LustreError: 303596:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.403661] LustreError: 303596:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [10952.943940] loop1: detected capacity change from 0 to 204800000 [10959.273873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.597375] Lustre: fs5861-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.191347] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [10980.373103] loop2: detected capacity change from 0 to 204800000 [10988.824962] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10999.367697] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11010.853340] Lustre: server umount fs5861-OST0000 complete [11020.562848] Lustre: server umount fs5861-OST0001 complete [11026.436043] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11034.595788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11034.607104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.628700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.640800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.646627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.667796] Lustre: Skipped 1 previous similar message [11039.779388] Lustre: server umount lustre-OST0000 complete [11046.896418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11046.898568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11046.923780] Lustre: Skipped 1 previous similar message [11046.933075] Lustre: Skipped 1 previous similar message [11052.001395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11052.005878] Lustre: Skipped 1 previous similar message [11056.104442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11058.144437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11058.369807] Lustre: server umount lustre-MDT0000 complete [11062.673415] LustreError: 301745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852240 with bad export cookie 361969691362918446 [11062.677801] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11062.682133] LustreError: 301745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11063.179331] Lustre: server umount lustre-MDT0001 complete [11076.914527] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [11079.791675] Key type lgssc unregistered [11080.143511] LNet: 315024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11080.156988] LNetError: 315024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11080.172634] LNet: Removed LNI 192.168.206.118@tcp [11081.232139] Key type .llcrypt unregistered [11081.233866] Key type ._llcrypt unregistered [11114.923497] Key type ._llcrypt registered [11114.926646] Key type .llcrypt registered [11115.045503] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [11130.415205] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [11131.627735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11131.686749] alg: No test for adler32 (adler32-zlib) [11132.753413] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11133.013518] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [11134.711174] Key type lgssc registered [11135.996704] Lustre: Echo OBD driver; http://www.lustre.org/ [11173.295851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11173.318555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.496567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11174.515483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11174.583692] Lustre: lustre-MDT0000: new disk, initializing [11174.641333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11174.670428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11179.557968] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11192.213540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.345717] Lustre: 318886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11192.404076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11192.408128] Lustre: Skipped 1 previous similar message [11192.474328] Lustre: lustre-MDT0001: new disk, initializing [11192.612473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11192.672625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11192.693205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11197.612913] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11209.746179] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11217.461996] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11227.821073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.174193] Lustre: lustre-OST0000: new disk, initializing [11228.184575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11228.195050] Lustre: 320856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11228.358966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.934074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11234.953438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11235.178916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11235.862119] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11247.880760] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11250.356807] Lustre: Mounted lustre-client - version 2.17.57_95_g5b39979 [11251.548889] Lustre: Unmounted lustre-client [11266.017961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11266.020617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11266.040968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11266.050841] Lustre: Skipped 1 previous similar message [11271.142899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.153058] Lustre: Skipped 1 previous similar message [11276.255133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11276.263827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11276.277428] Lustre: Skipped 1 previous similar message [11276.455660] Lustre: server umount lustre-OST0000 complete [11284.961905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11284.969744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.992726] Lustre: Skipped 1 previous similar message [11284.999963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11286.499139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.072755] Lustre: server umount lustre-MDT0000 complete [11292.703047] LustreError: 318877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852470 with bad export cookie 8692859404742932274 [11292.703842] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11292.713240] LustreError: 318877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11293.188748] Lustre: server umount lustre-MDT0001 complete [11303.010363] LNet: 1 local NIs in recovery (showing 1): 192.168.206.118@tcp [11305.316056] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [11309.367989] Key type lgssc unregistered [11309.883656] LNet: 323021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11309.893282] LNetError: 323021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11380.587728] LNet: Removed LNI 192.168.206.118@tcp [11382.460515] Key type .llcrypt unregistered [11382.469598] Key type ._llcrypt unregistered [11408.025600] Key type ._llcrypt registered [11408.028838] Key type .llcrypt registered [11408.162952] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [11424.931796] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [11426.142277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11426.183212] alg: No test for adler32 (adler32-zlib) [11427.569931] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11427.834731] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [11429.484060] Key type lgssc registered [11430.352768] Lustre: Echo OBD driver; http://www.lustre.org/ [11466.657772] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:44:01 (1787852641) [11475.440662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11475.525374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.042634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11477.131237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11477.228043] Lustre: lustre-MDT0000: new disk, initializing [11477.360173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.404169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11482.367772] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11495.907605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.091493] Lustre: 326246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11496.151501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11496.157770] Lustre: Skipped 1 previous similar message [11496.267864] Lustre: lustre-MDT0001: new disk, initializing [11496.385278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11496.482299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11496.504271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11501.941304] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11513.164302] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11520.192317] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11527.476779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11527.994275] Lustre: lustre-OST0000: new disk, initializing [11528.006568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11528.020057] Lustre: 328214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11528.160750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11533.945491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11533.954363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11534.091992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11536.383417] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11546.342215] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11565.026370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11565.054937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.073915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11567.018643] Lustre: server umount lustre-OST0000 complete [11573.225503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11573.250582] LustreError: Skipped 1 previous similar message [11573.266417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11573.285039] Lustre: Skipped 1 previous similar message [11573.288711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11573.301250] Lustre: Skipped 1 previous similar message [11575.265322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11575.267739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.082620] Lustre: server umount lustre-MDT0000 complete [11580.591686] LustreError: 326239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852758 with bad export cookie 11206459065341440541 [11580.605174] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.616486] LustreError: 326239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11580.914834] Lustre: server umount lustre-MDT0001 complete [11592.737685] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [11595.680694] Key type lgssc unregistered [11595.967685] LNet: 330125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11595.976217] LNetError: 330125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11595.993780] LNet: Removed LNI 192.168.206.118@tcp [11597.017113] Key type .llcrypt unregistered [11597.018881] Key type ._llcrypt unregistered [11619.977853] Key type ._llcrypt registered [11619.980514] Key type .llcrypt registered [11620.062872] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [11621.341734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11621.438921] alg: No test for adler32 (adler32-zlib) [11622.436202] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11622.633926] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [11624.287171] Key type lgssc registered [11625.271300] Lustre: Echo OBD driver; http://www.lustre.org/ [11633.246124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11633.271872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.787056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11639.030950] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11644.917635] LustreError: 331401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11644.951139] LustreError: 331401:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11649.296883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.821627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11656.101269] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11664.700657] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11671.735137] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11678.376358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.861850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11685.246442] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11693.874089] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11703.873639] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11705.432902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11709.920671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.923278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.940195] Lustre: Skipped 2 previous similar messages [11709.953484] Lustre: Skipped 2 previous similar messages [11712.064696] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.18@tcp (stopping) [11715.043950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11715.060153] Lustre: Skipped 2 previous similar messages [11715.653645] Lustre: server umount lustre-MDT0000 complete [11717.165541] LustreError: 333241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.645156] LustreError: 331381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852897 with bad export cookie 18196374802851374052 [11719.646768] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11719.661540] LustreError: 331381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11720.164767] LustreError: 331398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.166560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.212381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11722.284907] LustreError: 331396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11726.152637] Lustre: server umount lustre-MDT0001 complete [11730.167944] Lustre: server umount lustre-OST0000 complete [11738.212757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.889595] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11744.497865] Lustre: 335581:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.118@tcp [11746.483273] Lustre: server umount MGS complete [11754.113946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.608115] LustreError: 336152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.626335] LustreError: 336152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11754.681109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11758.797143] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11764.199040] LustreError: 336152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.225376] LustreError: 336152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11767.731214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.057939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.678879] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11781.306338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11781.699206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11786.874505] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11793.699548] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11795.180597] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11800.192418] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11805.787435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [11811.439376] Lustre: server umount lustre-OST0000 complete [11819.488618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11819.494496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.497135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.534934] Lustre: Skipped 3 previous similar messages [11827.724584] Lustre: server umount lustre-MDT0000 complete [11829.729356] LustreError: 337168:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11829.735306] Lustre: 330711:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852991/real 1787852991] req@00000000d58d0566 x1874699537674112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787853007 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11829.748687] LustreError: 337168:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11829.789991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11829.802148] Lustre: Skipped 1 previous similar message [11830.754092] Lustre: 330712:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852992/real 1787852992] req@000000008433a5c9 x1874699537674880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787853008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11831.078525] LustreError: 336133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853009 with bad export cookie 18196374802851374892 [11831.082236] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11831.086065] LustreError: 336133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11831.408874] Lustre: server umount lustre-MDT0001 complete [11841.154158] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [11844.016208] Key type lgssc unregistered [11844.314882] LNet: 339591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11844.318675] LNetError: 339591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11844.339749] LNet: Removed LNI 192.168.206.118@tcp [11845.230939] Key type .llcrypt unregistered [11845.232555] Key type ._llcrypt unregistered [11863.264727] Key type ._llcrypt registered [11863.268084] Key type .llcrypt registered [11863.444051] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [11884.685900] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [11886.136664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11886.210718] alg: No test for adler32 (adler32-zlib) [11887.327598] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11887.710749] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [11889.519156] Key type lgssc registered [11890.485123] Lustre: Echo OBD driver; http://www.lustre.org/ [11925.419633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11925.442489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.706202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11926.743215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11926.870696] Lustre: lustre-MDT0000: new disk, initializing [11926.937729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11926.962828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11930.705250] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11941.650313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.809548] Lustre: 342751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11941.892245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11941.900023] Lustre: Skipped 1 previous similar message [11941.993687] Lustre: lustre-MDT0001: new disk, initializing [11942.082995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.107477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11942.113406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11946.658921] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11957.096115] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11963.784277] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11971.398950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11971.828962] Lustre: lustre-OST0000: new disk, initializing [11971.833237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11971.845731] Lustre: 344720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11971.962744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11978.345169] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [11981.420494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11981.434284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.537925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11987.732759] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11992.749958] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11992.946365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11997.389263] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11997.561550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12002.272613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12002.277021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.285315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.296582] Lustre: Skipped 1 previous similar message [12005.879511] Lustre: server umount lustre-OST0000 complete [12012.514213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.516851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.522852] Lustre: Skipped 1 previous similar message [12012.533607] Lustre: Skipped 2 previous similar messages [12015.734849] Lustre: server umount lustre-MDT0000 complete [12018.826469] LustreError: 346263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853196 with bad export cookie 2007879195004559883 [12018.829800] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12018.840250] LustreError: 346263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12019.158352] Lustre: server umount lustre-MDT0001 complete [12027.420465] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:53:23 (1787853203) [12043.876193] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [12054.475205] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [12090.187125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.366565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12090.397345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12090.474581] Lustre: lustre-MDT0000: new disk, initializing [12090.538033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.563833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12094.772534] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12105.601799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12105.776346] Lustre: 349482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12105.829103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12105.832377] Lustre: Skipped 1 previous similar message [12105.886458] Lustre: lustre-MDT0001: new disk, initializing [12105.961066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.983373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12105.992338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12110.598701] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12120.800105] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12128.430647] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12138.228685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.493204] Lustre: lustre-OST0000: new disk, initializing [12138.497434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12138.503342] Lustre: 351449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12138.600740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12144.811938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12144.825977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12144.959737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12147.502355] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12158.365066] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12196.321209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12196.322157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.335222] LustreError: Skipped 1 previous similar message [12196.335974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.345868] Lustre: Skipped 1 previous similar message [12196.364266] Lustre: Skipped 1 previous similar message [12198.848896] Lustre: server umount lustre-OST0000 complete [12203.496813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12203.503779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12203.523670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12208.385966] Lustre: server umount lustre-MDT0000 complete [12211.681366] LustreError: 352494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.703851] LustreError: 352494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12212.088139] LustreError: 349474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853390 with bad export cookie 2007879195004560772 [12212.103204] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12212.598714] Lustre: server umount lustre-MDT0001 complete [12222.669566] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12225.501738] Key type lgssc unregistered [12225.738662] LNet: 353626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12225.749629] LNetError: 353626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12225.776518] LNet: Removed LNI 192.168.206.118@tcp [12226.643131] Key type .llcrypt unregistered [12226.646541] Key type ._llcrypt unregistered [12313.627318] Key type ._llcrypt registered [12313.629826] Key type .llcrypt registered [12313.764753] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12314.136386] Key type .llcrypt unregistered [12314.141160] Key type ._llcrypt unregistered [12390.089302] Key type ._llcrypt registered [12390.091798] Key type .llcrypt registered [12390.189882] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12390.513505] Key type .llcrypt unregistered [12390.516113] Key type ._llcrypt unregistered [12416.343791] Key type ._llcrypt registered [12416.346294] Key type .llcrypt registered [12416.436427] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12416.671712] Key type .llcrypt unregistered [12416.673572] Key type ._llcrypt unregistered [12444.170707] Key type ._llcrypt registered [12444.174138] Key type .llcrypt registered [12444.258596] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12444.542279] Key type .llcrypt unregistered [12444.548103] Key type ._llcrypt unregistered [12520.006062] Key type ._llcrypt registered [12520.008376] Key type .llcrypt registered [12520.154992] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12520.500370] Key type .llcrypt unregistered [12520.504960] Key type ._llcrypt unregistered [12547.679988] Key type ._llcrypt registered [12547.685757] Key type .llcrypt registered [12547.780491] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12548.112337] Key type .llcrypt unregistered [12548.119566] Key type ._llcrypt unregistered [12573.041953] Key type ._llcrypt registered [12573.043789] Key type .llcrypt registered [12573.130487] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12573.393379] Key type .llcrypt unregistered [12573.396773] Key type ._llcrypt unregistered [12598.390868] Key type ._llcrypt registered [12598.393413] Key type .llcrypt registered [12598.477378] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12598.818444] Key type .llcrypt unregistered [12598.822583] Key type ._llcrypt unregistered [12623.744797] Key type ._llcrypt registered [12623.746857] Key type .llcrypt registered [12623.837894] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12624.091308] Key type .llcrypt unregistered [12624.092887] Key type ._llcrypt unregistered [12653.139537] Key type ._llcrypt registered [12653.143494] Key type .llcrypt registered [12653.312085] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12653.765358] Key type .llcrypt unregistered [12653.770266] Key type ._llcrypt unregistered [12722.700848] Key type ._llcrypt registered [12722.706245] Key type .llcrypt registered [12722.884132] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12723.386206] Key type .llcrypt unregistered [12723.394834] Key type ._llcrypt unregistered [12757.018436] Key type ._llcrypt registered [12757.021952] Key type .llcrypt registered [12757.196673] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12757.670208] Key type .llcrypt unregistered [12757.671819] Key type ._llcrypt unregistered [12786.268863] Key type ._llcrypt registered [12786.271597] Key type .llcrypt registered [12786.381223] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [12786.658558] Key type .llcrypt unregistered [12786.661713] Key type ._llcrypt unregistered [12839.711228] Key type ._llcrypt registered [12839.717054] Key type .llcrypt registered [12839.889480] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [12856.824793] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [12858.047426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12858.066060] alg: No test for adler32 (adler32-zlib) [12859.216950] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [12859.466108] LNet: Added LNI 192.168.206.118@tcp [8/256/0/180] [12861.159968] Key type lgssc registered [12862.391680] Lustre: Echo OBD driver; http://www.lustre.org/ [12902.891946] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12902.943639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12904.289296] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12904.317081] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12904.412234] Lustre: tst250fs-MDT0000: new disk, initializing [12904.501787] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12904.524516] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12910.357373] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12924.232512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12924.549640] Lustre: 373912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12924.626700] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12924.629712] Lustre: Skipped 1 previous similar message [12924.789778] Lustre: tst250fs-MDT0001: new disk, initializing [12924.973885] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12925.065741] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12925.093609] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12931.970111] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12937.563115] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12946.290815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12946.605922] Lustre: tst250fs-OST0000: new disk, initializing [12946.610814] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12946.617040] Lustre: 375546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12946.764691] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12954.959186] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12955.752095] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12955.771571] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12955.904211] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12966.145913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12966.333543] Lustre: tst250fs-OST0001: new disk, initializing [12966.337700] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12966.343481] Lustre: 376416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12966.427443] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12972.975800] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_default_debug -1 all [12974.755801] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12974.786566] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12974.892103] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12978.755742] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12980.907234] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12980.915238] Lustre: Skipped 1 previous similar message [13000.673631] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.682789] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13000.691092] Lustre: Skipped 2 previous similar messages [13000.694823] Lustre: Skipped 1 previous similar message [13005.795086] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13005.809535] Lustre: Skipped 5 previous similar messages [13007.016734] Lustre: server umount tst250fs-MDT0000 complete [13010.770576] LustreError: 373905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854188 with bad export cookie 13920607421070027653 [13010.776447] LustreError: MGC192.168.206.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13010.782435] LustreError: 373905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13010.912780] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13010.914959] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [13010.935697] Lustre: Skipped 2 previous similar messages [13010.957578] Lustre: Skipped 1 previous similar message [13011.110277] Lustre: server umount tst250fs-MDT0001 complete [13018.037557] Lustre: server umount tst250fs-OST0000 complete [13021.667712] Lustre: server umount tst250fs-OST0001 complete [13043.859074] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [13053.572175] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [13086.386489] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:11:01 (1787854261) [13088.341788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13107.607878] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing set_hostid [13118.277639] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing load_modules_local [13151.146251] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12858 sec ========= 14:12:06 (1787854326) [13152.500335] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:12:08 (1787854328) === [13156.124292] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:12:11 (1787854331) === [13199.420390] Lustre: DEBUG MARKER: oleg618-server.virtnet: executing unload_modules_local [13202.408847] Key type lgssc unregistered [13202.771579] LNet: 385550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13202.776896] LNetError: 385550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13203.814789] LNet: Removed LNI 192.168.206.118@tcp [13204.992240] Key type .llcrypt unregistered [13204.994155] Key type ._llcrypt unregistered