[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525102787 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003022] Switched APIC routing to physical x2apic. [ 0.004039] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011101] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022335] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026698] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028063] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031449] *** VALIDATE proc *** [ 0.032226] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.040983] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.044172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045888] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.054202] rcu: Hierarchical SRCU implementation. [ 0.056560] smp: Bringing up secondary CPUs ... [ 0.057618] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062250] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234030] node 0 deferred pages initialised in 166ms [ 0.238228] devtmpfs: initialized [ 0.239235] x86/mm: Memory block size: 128MB [ 0.242918] gcov: version magic: 0x41383552 [ 0.245359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252389] pinctrl core: initialized pinctrl subsystem [ 0.254270] [ 0.254862] ************************************************************* [ 0.257020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260019] ** ** [ 0.262017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264017] ** ** [ 0.267017] ** This means that this kernel is built to expose internal ** [ 0.269014] ** IOMMU data structures, which may compromise security on ** [ 0.271019] ** your system. ** [ 0.273018] ** ** [ 0.276017] ** If you see this message and you are not debugging the ** [ 0.278016] ** kernel, report this immediately to your vendor! ** [ 0.281018] ** ** [ 0.283014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285022] ************************************************************* [ 0.288767] NET: Registered protocol family 16 [ 0.290492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299040] cpuidle: using governor menu [ 0.301207] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304764] PCI: Using configuration type 1 for base access [ 0.306148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.316186] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.317021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319175] cryptd: max_cpu_qlen set to 1000 [ 0.322358] ACPI: Added _OSI(Module Device) [ 0.324041] ACPI: Added _OSI(Processor Device) [ 0.325022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.327015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.337439] ACPI: Interpreter enabled [ 0.339080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.340018] ACPI: Using IOAPIC for interrupt routing [ 0.342106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.346431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356624] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.358054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.361025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370753] acpiphp: Slot [2] registered [ 0.372181] acpiphp: Slot [3] registered [ 0.373099] acpiphp: Slot [4] registered [ 0.375092] acpiphp: Slot [5] registered [ 0.376127] acpiphp: Slot [6] registered [ 0.377137] acpiphp: Slot [7] registered [ 0.379166] acpiphp: Slot [8] registered [ 0.380341] acpiphp: Slot [9] registered [ 0.382189] acpiphp: Slot [10] registered [ 0.383156] acpiphp: Slot [11] registered [ 0.385153] acpiphp: Slot [12] registered [ 0.387140] acpiphp: Slot [13] registered [ 0.389151] acpiphp: Slot [14] registered [ 0.390107] acpiphp: Slot [15] registered [ 0.392118] acpiphp: Slot [16] registered [ 0.393092] acpiphp: Slot [17] registered [ 0.394116] acpiphp: Slot [18] registered [ 0.396105] acpiphp: Slot [19] registered [ 0.397134] acpiphp: Slot [20] registered [ 0.399150] acpiphp: Slot [21] registered [ 0.400177] acpiphp: Slot [22] registered [ 0.402121] acpiphp: Slot [23] registered [ 0.403118] acpiphp: Slot [24] registered [ 0.405132] acpiphp: Slot [25] registered [ 0.406141] acpiphp: Slot [26] registered [ 0.408146] acpiphp: Slot [27] registered [ 0.409129] acpiphp: Slot [28] registered [ 0.411166] acpiphp: Slot [29] registered [ 0.412130] acpiphp: Slot [30] registered [ 0.414129] acpiphp: Slot [31] registered [ 0.415077] PCI host bridge to bus 0000:00 [ 0.417047] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.427026] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.429028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.430142] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.434772] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.439733] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451908] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.459081] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.461021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.464024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.466021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.469592] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.472847] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.475055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.478635] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.483020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.497021] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.503022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509183] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.515023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.522019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.540022] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.551172] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.561024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.582023] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.592417] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.633966] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.640021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.648029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.672369] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.685017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703020] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.713714] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.720018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.729020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747014] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.759752] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.763605] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.766422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.769393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.771241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.776186] iommu: Default domain type: Passthrough [ 0.778475] SCSI subsystem initialized [ 0.780147] ACPI: bus type USB registered [ 0.781101] usbcore: registered new interface driver usbfs [ 0.783117] usbcore: registered new interface driver hub [ 0.785121] usbcore: registered new device driver usb [ 0.787177] pps_core: LinuxPPS API ver. 1 registered [ 0.789013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.792066] PTP clock support registered [ 0.795053] EDAC MC: Ver: 3.0.0 [ 0.796327] PCI: Using ACPI for IRQ routing [ 0.800273] NetLabel: Initializing [ 0.802016] NetLabel: domain hash size = 128 [ 0.803011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.805084] NetLabel: unlabeled traffic allowed by default [ 0.809072] vgaarb: loaded [ 0.811377] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.812015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.821986] clocksource: Switched to clocksource kvm-clock [ 0.929568] VFS: Disk quotas dquot_6.6.0 [ 0.931464] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934132] *** VALIDATE ramfs *** [ 0.935479] *** VALIDATE hugetlbfs *** [ 0.937120] pnp: PnP ACPI init [ 0.939635] pnp: PnP ACPI: found 6 devices [ 0.958439] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.961989] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.964103] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.966196] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.968703] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.971194] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.973835] NET: Registered protocol family 2 [ 0.976399] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.980984] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.984170] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.989467] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.993185] TCP: Hash tables configured (established 65536 bind 65536) [ 0.996231] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.999331] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002238] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005280] NET: Registered protocol family 1 [ 1.008331] RPC: Registered named UNIX socket transport module. [ 1.010572] RPC: Registered udp transport module. [ 1.012491] RPC: Registered tcp transport module. [ 1.014218] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.016071] NET: Registered protocol family 44 [ 1.017812] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.019952] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.022574] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.025159] PCI: CLS 0 bytes, default 64 [ 1.026590] Unpacking initramfs... [ 2.444976] debug: unmapping init [mem 0xffff88ea7cc54000-0xffff88ea7ffbffff] [ 2.449019] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.451575] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.454106] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.946705] Initialise system trusted keyrings [ 2.948106] Key type blacklist registered [ 2.949700] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.959251] zbud: loaded [ 2.962483] *** VALIDATE nfs *** [ 2.963623] *** VALIDATE nfs4 *** [ 2.966280] pstore: using deflate compression [ 2.969835] Platform Keyring initialized [ 3.088257] NET: Registered protocol family 38 [ 3.090353] Key type asymmetric registered [ 3.091855] Asymmetric key parser 'x509' registered [ 3.093710] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.096845] io scheduler mq-deadline registered [ 3.098687] io scheduler kyber registered [ 3.100407] io scheduler bfq registered [ 3.102273] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.105225] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.108134] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.111099] ACPI: Power Button [PWRF] [ 3.202299] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.296248] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.485856] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.581970] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.778964] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.808465] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.837067] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.843491] Non-volatile memory driver v1.3 [ 3.845368] Linux agpgart interface v0.103 [ 3.876685] virtio_blk virtio1: [vda] 133624 512-byte logical blocks (68.4 MB/65.2 MiB) [ 3.879923] vda: detected capacity change from 0 to 68415488 [ 3.896066] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.899175] vdb: detected capacity change from 0 to 1073741824 [ 3.912949] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.916371] vdc: detected capacity change from 0 to 2621440000 [ 3.929790] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.932716] vdd: detected capacity change from 0 to 2621440000 [ 3.947441] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.950513] vde: detected capacity change from 0 to 4294967296 [ 3.963584] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.966381] vdf: detected capacity change from 0 to 4294967296 [ 3.972039] libphy: Fixed MDIO Bus: probed [ 3.979724] usbcore: registered new interface driver usbserial_generic [ 3.982411] usbserial: USB Serial support registered for generic [ 3.985026] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.989349] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.991168] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.993655] mousedev: PS/2 mouse device common for all mice [ 3.996418] rtc_cmos 00:05: RTC can wake from S4 [ 3.998589] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.001973] rtc_cmos 00:05: registered as rtc0 [ 4.004877] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.006239] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.012981] intel_pstate: CPU model not supported [ 4.013422] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.019638] hid: raw HID events driver (C) Jiri Kosina [ 4.022091] usbcore: registered new interface driver usbhid [ 4.023733] usbhid: USB HID core driver [ 4.025092] drop_monitor: Initializing network drop monitor service [ 4.026919] Initializing XFRM netlink socket [ 4.028869] NET: Registered protocol family 10 [ 4.032160] Segment Routing with IPv6 [ 4.033419] NET: Registered protocol family 17 [ 4.035433] mpls_gso: MPLS GSO support [ 4.040667] RAS: Correctable Errors collector initialized. [ 4.042669] AVX version of gcm_enc/dec engaged. [ 4.044360] AES CTR mode by8 optimization enabled [ 4.119712] sched_clock: Marking stable (4119679836, 0)->(5062251691, -942571855) [ 4.123475] registered taskstats version 1 [ 4.125825] Loading compiled-in X.509 certificates [ 4.127491] zswap: loaded using pool lzo/zbud [ 4.152908] Key type big_key registered [ 4.165412] Key type encrypted registered [ 4.166949] ima: No TPM chip found, activating TPM-bypass! [ 4.168823] ima: Allocated hash algorithm: sha1 [ 4.170163] ima: No architecture policies found [ 4.171996] evm: Initialising EVM extended attributes: [ 4.173886] evm: security.selinux [ 4.175250] evm: security.ima [ 4.176438] evm: security.capability [ 4.177852] evm: HMAC attrs: 0x1 [ 4.180481] rtc_cmos 00:05: setting system clock to 2025-11-14 00:27:50 UTC (1763080070) [ 4.187563] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 4.190921] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 4.200661] Write protecting the kernel read-only data: 28672k [ 4.203991] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 4.206683] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 4.239749] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.247119] systemd[1]: Detected virtualization kvm. [ 4.248730] systemd[1]: Detected architecture x86-64. [ 4.250307] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.278295] systemd[1]: No hostname configured. [ 4.279935] systemd[1]: Set hostname to . [ 4.281633] random: systemd: uninitialized urandom read (16 bytes read) [ 4.283463] systemd[1]: Initializing machine ID from random generator. [ 4.323992] random: ln: uninitialized urandom read (6 bytes read) [ 4.389346] random: systemd: uninitialized urandom read (16 bytes read) [ 4.392091] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.397068] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.400973] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.973613] device-mapper: uevent: version 1.0.3 [ 4.975884] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.601534] virtio_net virtio0 ens2: renamed from eth0 [ 5.615813] random: fast init done [ 5.650844] scsi host0: ata_piix [ 5.675508] scsi host1: ata_piix [ 5.676983] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.679120] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.044067] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.686520] random: crng init done [ 10.687920] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.980322] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.136105] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.390088] SELinux: Disabled at runtime. [ 12.452801] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.460476] systemd[1]: Detected virtualization kvm. [ 12.461985] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.968227] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.972123] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.978242] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.982359] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.985680] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.993156] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.000943] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 13.060086] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.485313] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.829699] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.852293] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.005329] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.023680] EDAC sbridge: Ver: 1.1.2 [ 15.573565] Key type dns_resolver registered [ 15.876341] NFS: Registering the id_resolver key type [ 15.878034] Key type id_resolver registered [ 15.879651] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 65.459606] libcfs: loading out-of-tree module taints kernel. [ 65.483601] Key type ._llcrypt registered [ 65.485795] Key type .llcrypt registered [ 65.564181] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 77.860677] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 78.908244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.919943] alg: No test for adler32 (adler32-zlib) [ 80.229174] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 81.058487] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 82.752149] Key type lgssc registered [ 84.067355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.574765] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.163884] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.573987] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.858697] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.640311] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.424775] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 133.017754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.078500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.101996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.314173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.339245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.402435] Lustre: lustre-MDT0000: new disk, initializing [ 134.475043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.489345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.493935] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.985015] hrtimer: interrupt took 7548379 ns [ 148.023743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.090399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 148.190948] Lustre: 6506:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.241655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.244554] Lustre: Skipped 1 previous similar message [ 148.335060] Lustre: lustre-MDT0001: new disk, initializing [ 148.412139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.446906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.453402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 151.437843] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.925162] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 161.304656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.399708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 161.634726] Lustre: lustre-OST0000: new disk, initializing [ 161.638509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 161.674372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.594710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 162.605375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 162.690602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 166.438687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.483022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.549277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 180.661755] Lustre: lustre-OST0001: new disk, initializing [ 180.667844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 180.737637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.948033] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.452985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 188.469536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 188.506363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 196.694547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.140391] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 212.052232] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 215.695552] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 219.361570] Lustre: DEBUG MARKER: Client: 2.16.61.1 [ 221.305184] Lustre: DEBUG MARKER: MDS: 2.16.61.1 [ 223.251446] Lustre: DEBUG MARKER: OSS: 2.16.61.1 [ 224.329303] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Nov 13 19:31:29 EST 2025 [ 236.819900] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 238.173321] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 254.947716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.948446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.954856] Lustre: Skipped 3 previous similar messages [ 254.961102] Lustre: Skipped 3 previous similar messages [ 257.711481] LustreError: 12211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 257.788765] Lustre: server umount lustre-MDT0000 complete [ 260.227884] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080326 with bad export cookie 12019660505971210713 [ 260.234972] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 260.237031] LustreError: 6498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 265.185979] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.186213] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.187984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.203495] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 265.208180] Lustre: Skipped 1 previous similar message [ 266.490054] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 266.538692] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 266.664669] Lustre: server umount lustre-MDT0001 complete [ 269.582351] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 269.597801] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 269.656234] Lustre: server umount lustre-OST0000 complete [ 272.139761] LustreError: 12814:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 272.142017] LustreError: 12814:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 272.222474] Lustre: server umount lustre-OST0001 complete [ 277.160870] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 284.005596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 291.639353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 296.795613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 302.310785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.293337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.096941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.162936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.365674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 315.396646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 315.455514] Lustre: lustre-MDT0000: new disk, initializing [ 315.552962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 315.575697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 319.140113] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 328.504475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.591576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 328.689878] Lustre: 14996:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 328.739311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 328.743089] Lustre: Skipped 1 previous similar message [ 328.823317] Lustre: lustre-MDT0001: new disk, initializing [ 328.943614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 329.028596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 329.039222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 331.907844] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 337.819427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 340.511641] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 345.653785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.721871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.925985] Lustre: lustre-OST0000: new disk, initializing [ 345.931585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 346.013559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.381369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 347.390065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 347.492226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 349.853954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 355.564148] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 358.623800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 358.802283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 361.463831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 361.637370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 362.981404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 362.982164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.989206] LustreError: Skipped 1 previous similar message [ 362.992168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 363.001053] Lustre: Skipped 1 previous similar message [ 363.005253] Lustre: Skipped 1 previous similar message [ 368.098200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 368.108297] Lustre: Skipped 1 previous similar message [ 373.222584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 373.229422] Lustre: Skipped 1 previous similar message [ 377.312071] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.343326] LustreError: 17937:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 377.346412] LustreError: 17937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 377.436337] Lustre: server umount lustre-OST0000 complete [ 380.386257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 380.393131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.460907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.464649] Lustre: Skipped 2 previous similar messages [ 386.251394] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 386.256900] LustreError: 18141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 386.320608] Lustre: server umount lustre-MDT0000 complete [ 388.583085] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.592842] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 388.777952] LustreError: 14988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080455 with bad export cookie 12019660505971212246 [ 388.781233] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.787280] LustreError: 14988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 389.093657] Lustre: server umount lustre-MDT0001 complete [ 393.235672] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:34:18 (1763080458) [ 405.458170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.795374] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.863340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.761308] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 411.105681] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.220771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 418.759234] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 422.897597] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.728421] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 427.293148] Lustre: Failing over lustre-MDT0000 [ 427.300863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.310464] Lustre: Skipped 2 previous similar messages [ 427.311047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.320539] Lustre: Skipped 1 previous similar message [ 427.403037] LustreError: 20818:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 427.410875] LustreError: 20818:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 427.473217] Lustre: server umount lustre-MDT0000 complete [ 432.610129] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.641489] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 440.802542] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.819038] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 443.844378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.966733] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 447.520515] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 449.509915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 449.510542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 449.543334] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 454.570650] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 459.746870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.747848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.760993] Lustre: Skipped 1 previous similar message [ 459.765364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.521068] LustreError: 21798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 464.524367] LustreError: 21798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 464.651975] Lustre: server umount lustre-MDT0000 complete [ 465.897251] LustreError: 19475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.910869] LustreError: 19475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 468.253061] LustreError: 19457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080534 with bad export cookie 12019660505971213660 [ 468.266332] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.721725] Lustre: server umount lustre-MDT0001 complete [ 475.807595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.242413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.245854] Lustre: Skipped 2 previous similar messages [ 479.419447] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 486.181346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 488.847336] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 492.027876] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 494.406390] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.912731] Lustre: Failing over lustre-MDT0000 [ 495.929170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.954809] Lustre: Skipped 1 previous similar message [ 496.108923] Lustre: server umount lustre-MDT0000 complete [ 502.242139] LustreError: 23587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.258421] LustreError: 23587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 512.621721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.694318] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.356890] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 518.119140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 518.120466] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 518.129975] Lustre: Skipped 1 previous similar message [ 518.143920] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 520.757644] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 528.354904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.356153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.368424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.371522] Lustre: Skipped 1 previous similar message [ 528.378439] Lustre: Skipped 7 previous similar messages [ 530.119937] LustreError: 24795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 530.127772] LustreError: 24795:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 530.255317] Lustre: server umount lustre-MDT0000 complete [ 533.334390] LustreError: 22457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080599 with bad export cookie 12019660505971214591 [ 533.344288] LustreError: 22457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 533.344483] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.659520] Lustre: server umount lustre-MDT0001 complete [ 540.929490] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:36:46 (1763080606) [ 553.167344] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 562.302764] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 571.791466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 578.929155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 584.971336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 591.520746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 599.637923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 609.336882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 609.404339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.583077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 609.611612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 609.665449] Lustre: lustre-MDT0000: new disk, initializing [ 609.753714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.758361] Lustre: Skipped 2 previous similar messages [ 609.777331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 612.754469] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 622.448287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.550863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 622.723830] Lustre: 28468:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 622.979143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 626.113342] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 633.357972] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.916748] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 642.211427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 642.275839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 644.284863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 644.315673] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 644.408315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 644.447394] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 647.795811] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 654.111471] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 658.693525] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 675.299318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 675.299584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.302763] LustreError: Skipped 1 previous similar message [ 675.304849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 675.324480] Lustre: Skipped 1 previous similar message [ 689.121584] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 689.230071] LustreError: 31129:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 689.232527] LustreError: 31129:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 689.407991] Lustre: server umount lustre-OST0000 complete [ 694.757222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 699.545569] Lustre: server umount lustre-MDT0000 complete [ 700.898403] LustreError: 29369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.904845] LustreError: 29369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 702.338426] LustreError: 28461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080768 with bad export cookie 12019660505971214990 [ 702.347770] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.347937] LustreError: 28461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 702.686240] Lustre: server umount lustre-MDT0001 complete [ 711.193064] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 714.050963] Key type lgssc unregistered [ 714.432249] LNet: 32064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.443852] LNetError: 32064:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.483803] LNet: Removed LNI 192.168.203.145@tcp [ 715.460369] Key type .llcrypt unregistered [ 715.462375] Key type ._llcrypt unregistered [ 735.179976] Key type ._llcrypt registered [ 735.181594] Key type .llcrypt registered [ 735.276760] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 736.142923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.203657] alg: No test for adler32 (adler32-zlib) [ 737.253289] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 737.439052] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 739.072660] Key type lgssc registered [ 739.943604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 746.214892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 746.245203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.010191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.912823] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 758.250979] LustreError: 33437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.265850] LustreError: 33437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 759.259799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 759.577123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 762.663641] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 767.193678] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 770.266365] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 776.535911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 776.884135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.967482] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 784.071630] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 787.232988] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 793.318827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 808.963874] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:41:13 (1763080873) [ 810.696550] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 812.376265] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 818.287571] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:41:23 (1763080883) [ 825.421597] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:41:30 (1763080890) [ 826.293229] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 834.011535] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 834.022780] Lustre: Skipped 2 previous similar messages [ 835.532860] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:41:40 (1763080900) [ 838.530577] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 838.532847] Lustre: Skipped 3 previous similar messages [ 854.425164] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 854.430397] Lustre: Skipped 6 previous similar messages [ 856.919743] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:42:01 (1763080921) [ 859.516124] Lustre: *** cfs_fail_loc=131b, val=2*** [ 862.688952] Lustre: *** cfs_fail_loc=131b, val=2*** [ 870.520823] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:42:15 (1763080935) [ 872.349618] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 872.352546] Lustre: Skipped 3 previous similar messages [ 885.678969] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:42:30 (1763080950) [ 901.092297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.105192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 901.107085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 903.712122] LustreError: 40028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 903.809712] Lustre: server umount lustre-OST0000 complete [ 908.256944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 908.264769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.277824] Lustre: Skipped 1 previous similar message [ 908.286068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.288834] Lustre: Skipped 1 previous similar message [ 911.329883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 911.341737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.779337] LustreError: 40231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 912.791860] LustreError: 40231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 912.956845] Lustre: server umount lustre-MDT0000 complete [ 916.450154] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.782264] LustreError: 33419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080983 with bad export cookie 8381918483571714291 [ 916.786608] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.828762] LustreError: 33419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 917.003874] LustreError: 40432:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 917.017464] LustreError: 40432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 917.281871] Lustre: server umount lustre-MDT0001 complete [ 926.582883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 928.903647] Key type lgssc unregistered [ 929.178221] LNet: 40964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.187488] LNetError: 40964:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.206723] LNet: Removed LNI 192.168.203.145@tcp [ 929.961831] Key type .llcrypt unregistered [ 929.964569] Key type ._llcrypt unregistered [ 945.947834] Key type ._llcrypt registered [ 945.949646] Key type .llcrypt registered [ 946.035781] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 959.778916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 960.599729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.689657] alg: No test for adler32 (adler32-zlib) [ 961.653854] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 961.837468] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 963.464411] Key type lgssc registered [ 964.361378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.688118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 977.607908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 983.156916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 989.197137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 995.287226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 995.337575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 995.349563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.537780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 996.565549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 996.634822] Lustre: lustre-MDT0000: new disk, initializing [ 996.727914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.742582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 999.726883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1008.209569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.271708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1008.350317] Lustre: 44094:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1008.407805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1008.415782] Lustre: Skipped 1 previous similar message [ 1008.490525] Lustre: lustre-MDT0001: new disk, initializing [ 1008.562399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.594581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1008.614621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1011.795917] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1018.688068] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1022.806704] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.790832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.861390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.188968] Lustre: lustre-OST0000: new disk, initializing [ 1030.196298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1030.314368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.491961] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1037.882699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1037.906822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1037.973118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1042.587228] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1046.536687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1046.765799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1050.688954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1050.928328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1053.666464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1053.680339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.688926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.718335] LustreError: Skipped 1 previous similar message [ 1053.731155] Lustre: Skipped 1 previous similar message [ 1058.750233] LustreError: 47032:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1058.879151] Lustre: server umount lustre-OST0000 complete [ 1063.907632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.919348] Lustre: Skipped 1 previous similar message [ 1063.923797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.026509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.040651] Lustre: Skipped 2 previous similar messages [ 1074.147258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.161514] Lustre: Skipped 1 previous similar message [ 1076.194333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1076.317166] LustreError: 47234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1076.320355] LustreError: 47234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1076.391725] Lustre: server umount lustre-MDT0000 complete [ 1079.265333] LustreError: 44106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.288846] LustreError: 44106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1079.723959] LustreError: 45839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081146 with bad export cookie 1637964690175231441 [ 1079.730090] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.739134] LustreError: 45839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1079.844941] LustreError: 47437:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1079.848083] LustreError: 47437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1079.985316] Lustre: server umount lustre-MDT0001 complete [ 1089.661544] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:45:54 (1763081154) [ 1095.438808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.886882] LustreError: 48155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.988414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.994892] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1101.283975] LustreError: 48155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.407430] LustreError: 48155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.143692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1108.567077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.048572] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1117.331962] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1121.447424] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1126.942925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.190338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.660181] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1136.977929] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1145.637166] LustreError: 50352:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1146.456533] Lustre: Modifying parameter general.timeout=1 in log params [ 1147.963760] Lustre: Modifying parameter general.timeout=3 in log params [ 1147.973780] Lustre: Skipped 1 previous similar message [ 1150.582826] Lustre: Modifying parameter general.timeout=6 in log params [ 1150.585755] Lustre: Skipped 2 previous similar messages [ 1155.095253] Lustre: Modifying parameter general.timeout=12 in log params [ 1155.099803] Lustre: Skipped 5 previous similar messages [ 1163.476392] Lustre: Modifying parameter general.timeout=23 in log params [ 1163.478612] Lustre: Skipped 10 previous similar messages [ 1179.532225] Lustre: Modifying parameter general.timeout=40 in log params [ 1179.535162] Lustre: Skipped 16 previous similar messages [ 1211.760865] Lustre: Modifying parameter general.timeout=85 in log params [ 1211.764583] Lustre: Skipped 44 previous similar messages [ 1250.180502] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:48:35 (1763081315) [ 1252.573955] Lustre: 57283:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1254.258681] Lustre: 57392:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1257.632512] Lustre: 57588:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1257.643462] Lustre: 57588:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1277.385228] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1277.396862] Lustre: Skipped 64 previous similar messages [ 1284.743636] Lustre: 59156:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1284.764648] Lustre: 59156:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1284.771132] LustreError: 59156:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1288.996279] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:49:13 (1763081353) [ 1303.010459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1303.025576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.040989] Lustre: Skipped 1 previous similar message [ 1303.046088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.397961] LustreError: 59856:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1305.402017] LustreError: 59856:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1305.485314] Lustre: server umount lustre-MDT0000 complete [ 1308.065291] LustreError: 50404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081374 with bad export cookie 1637964690175232316 [ 1308.074541] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.296510] Lustre: server umount lustre-MDT0001 complete [ 1322.186267] LustreError: 60258:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1322.201918] LustreError: 60258:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1322.288729] Lustre: server umount lustre-OST0000 complete [ 1328.317832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1331.413546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.336617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1337.483282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1342.892570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.100155] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1343.333989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.439771] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1355.010261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1355.132988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1355.167391] Lustre: 61710:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1359.589184] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1364.216266] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1368.221681] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1374.206549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1374.321787] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1378.835641] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1383.263731] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1384.549718] Lustre: 63940:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1390.050527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.056093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.076047] Lustre: Skipped 2 previous similar messages [ 1390.077545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.108099] Lustre: Skipped 4 previous similar messages [ 1392.026188] LustreError: 64195:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1392.028336] LustreError: 64195:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1392.230447] Lustre: server umount lustre-OST0000 complete [ 1396.712959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1396.723446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.733767] Lustre: Skipped 1 previous similar message [ 1402.155196] Lustre: server umount lustre-MDT0000 complete [ 1405.410301] LustreError: 63106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.419968] LustreError: 63106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1405.623156] LustreError: 64398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081471 with bad export cookie 1637964690175235193 [ 1405.631725] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.641533] LustreError: 64398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1405.959354] Lustre: server umount lustre-MDT0001 complete [ 1416.288681] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1418.827490] Key type lgssc unregistered [ 1419.174578] LNet: 65133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.190501] LNetError: 65133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.209680] LNet: Removed LNI 192.168.203.145@tcp [ 1420.000707] Key type .llcrypt unregistered [ 1420.006361] Key type ._llcrypt unregistered [ 1441.359835] Key type ._llcrypt registered [ 1441.361764] Key type .llcrypt registered [ 1441.458689] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1442.392969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.439155] alg: No test for adler32 (adler32-zlib) [ 1443.422549] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1443.615842] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1445.264167] Key type lgssc registered [ 1445.884883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1451.061734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1451.077759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.516407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.637145] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1461.730893] LustreError: 66523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.742751] LustreError: 66523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1462.509472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1462.951635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1465.591385] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1469.278297] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1471.791848] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1476.469351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1476.796606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.937386] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1483.151687] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1494.152639] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1503.715521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.717507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1503.731134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.739739] Lustre: Skipped 1 previous similar message [ 1506.429629] LustreError: 69231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1506.520213] Lustre: server umount lustre-MDT0000 complete [ 1508.835090] LustreError: 67640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.844557] LustreError: 67640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1509.218150] LustreError: 66504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081575 with bad export cookie 5604369478182135971 [ 1509.221901] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.230182] LustreError: 66504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1509.357900] LustreError: 69435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1509.363183] LustreError: 69435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1509.523412] Lustre: server umount lustre-MDT0001 complete [ 1521.799647] LustreError: 69636:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1521.803290] LustreError: 69636:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1521.881104] Lustre: server umount lustre-OST0000 complete [ 1527.839579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1530.309398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.512911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.708848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.986973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.237995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1539.273931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1539.474891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.060145] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1547.434067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1547.525526] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1547.554471] Lustre: 71086:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1547.865485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1550.421293] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1553.783312] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1556.239776] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1561.609793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1561.754286] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1562.182273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.385055] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1570.059191] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1571.466745] Lustre: 73328:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1572.961159] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1577.955523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.957683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.958304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.958309] Lustre: Skipped 2 previous similar messages [ 1577.972852] Lustre: Skipped 2 previous similar messages [ 1582.333254] LustreError: 73575:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1582.341431] LustreError: 73575:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1582.433387] Lustre: server umount lustre-OST0000 complete [ 1588.196109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.200295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.209560] Lustre: Skipped 1 previous similar message [ 1588.223124] Lustre: Skipped 2 previous similar messages [ 1591.323864] LustreError: 73785:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1591.362944] LustreError: 73785:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1591.528927] Lustre: server umount lustre-MDT0000 complete [ 1593.312869] LustreError: 71098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.322585] LustreError: 71098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1594.723601] LustreError: 71816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081661 with bad export cookie 5604369478182136797 [ 1594.730120] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.736849] LustreError: 71816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1594.997231] Lustre: server umount lustre-MDT0001 complete [ 1602.600876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1604.928616] Key type lgssc unregistered [ 1605.172726] LNet: 74519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.188106] LNetError: 74519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.204559] LNet: Removed LNI 192.168.203.145@tcp [ 1605.962778] Key type .llcrypt unregistered [ 1605.969635] Key type ._llcrypt unregistered [ 1625.650808] Key type ._llcrypt registered [ 1625.652887] Key type .llcrypt registered [ 1625.754432] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1626.696816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1626.739930] alg: No test for adler32 (adler32-zlib) [ 1627.793587] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1628.019862] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1629.683338] Key type lgssc registered [ 1630.728830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.842884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1637.880501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.379678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.413132] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1649.323807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1649.818760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.862412] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1656.281308] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1659.072593] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.030610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1665.734780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.613759] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1675.247221] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1690.393983] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1691.203948] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1692.758674] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1692.761701] Lustre: Skipped 1 previous similar message [ 1695.489617] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1695.494632] Lustre: Skipped 2 previous similar messages [ 1700.130358] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1700.135176] Lustre: Skipped 4 previous similar messages [ 1708.152351] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1708.156679] Lustre: Skipped 7 previous similar messages [ 1724.863978] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1724.867150] Lustre: Skipped 16 previous similar messages [ 1757.074376] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1757.077120] Lustre: Skipped 39 previous similar messages [ 1782.043894] Lustre: 83678:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1784.282246] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:57:29 (1763081849) [ 1785.447341] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1787.117427] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:57:32 (1763081852) [ 1791.968990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1791.972866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.988711] LustreError: Skipped 1 previous similar message [ 1791.990084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.002567] Lustre: Skipped 1 previous similar message [ 1797.093525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.096850] Lustre: Skipped 1 previous similar message [ 1802.209849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.212335] Lustre: Skipped 1 previous similar message [ 1804.256574] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1804.316474] LustreError: 83973:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1804.508770] Lustre: server umount lustre-OST0000 complete [ 1808.363559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.379263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.407582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.416207] Lustre: Skipped 1 previous similar message [ 1812.451752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.454774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.229033] LustreError: 84176:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1814.265685] LustreError: 84176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1814.381960] Lustre: server umount lustre-MDT0000 complete [ 1817.576097] LustreError: 75890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.598275] LustreError: 75890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1818.439982] LustreError: 79999:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081884 with bad export cookie 13952441539314675723 [ 1818.442241] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.449949] LustreError: 79999:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.666433] LustreError: 84377:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1818.668990] LustreError: 84377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1818.930897] Lustre: server umount lustre-MDT0001 complete [ 1828.426451] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1830.722612] Key type lgssc unregistered [ 1830.957437] LNet: 84909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.965462] LNetError: 84909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.987937] LNet: Removed LNI 192.168.203.145@tcp [ 1831.646421] Key type .llcrypt unregistered [ 1831.650545] Key type ._llcrypt unregistered [ 1835.141134] Key type ._llcrypt registered [ 1835.142884] Key type .llcrypt registered [ 1835.225109] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1838.296091] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules [ 1839.227847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.281731] alg: No test for adler32 (adler32-zlib) [ 1840.330581] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1840.589941] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1842.236146] Key type lgssc registered [ 1843.157911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.626743] LustreError: 86075:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1861.191216] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1867.279319] LustreError: 86075:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1871.053353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.071135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.511569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.407715] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1882.085069] LustreError: 87083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.099096] LustreError: 87083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1883.166388] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:59:08 (1763081948) [ 1887.204988] LustreError: 87083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.256684] LustreError: 87114:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1887.264376] LustreError: 87114:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1887.279612] LustreError: 87114:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 1887.389861] LustreError: 87889:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1887.570221] Lustre: server umount lustre-MDT0000 complete [ 1898.940119] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1901.238127] Key type lgssc unregistered [ 1901.470475] LNet: 88574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.478341] LNetError: 88574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.494696] LNet: Removed LNI 192.168.203.145@tcp [ 1902.138299] Key type .llcrypt unregistered [ 1902.142323] Key type ._llcrypt unregistered [ 1915.258588] Key type ._llcrypt registered [ 1915.259994] Key type .llcrypt registered [ 1915.400813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1915.981447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.046731] alg: No test for adler32 (adler32-zlib) [ 1917.024398] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1917.196667] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1918.856202] Key type lgssc registered [ 1919.769134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.974737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.988423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.391790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.467221] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1936.873552] LustreError: 89750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.886722] LustreError: 89750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1937.390348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.073607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1941.289831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1945.788187] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.037413] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1955.094487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.566161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.888770] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1964.250491] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2014.550519] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:01:19 (1763082079) [ 2015.966370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2017.598328] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:01:22 (1763082082) [ 2023.400955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.402045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.407054] Lustre: Skipped 1 previous similar message [ 2023.415687] Lustre: Skipped 1 previous similar message [ 2026.647532] LustreError: 92699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2026.730824] Lustre: server umount lustre-MDT0000 complete [ 2028.516735] LustreError: 90478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.538059] LustreError: 90478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2029.472277] LustreError: 89732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082095 with bad export cookie 16633879665319264205 [ 2029.482290] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.485841] LustreError: 89732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.622800] LustreError: 92902:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2029.626120] LustreError: 92902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2029.766820] Lustre: server umount lustre-MDT0001 complete [ 2043.139514] LustreError: 93103:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2043.142521] LustreError: 93103:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2043.460458] Lustre: server umount lustre-OST0000 complete [ 2054.241220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.708583] LustreError: 93733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.884309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.385551] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2060.267412] LustreError: 93733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.098930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2065.630720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.688860] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2072.313430] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2075.063492] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.758680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2085.255351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2085.323434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2085.477714] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2085.481865] LustreError: 93720:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2085.508874] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2085.515215] LustreError: 95593:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2085.519900] LustreError: 95593:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2085.523629] LustreError: 95593:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2085.580395] Lustre: server umount lustre-OST0000 complete [ 2085.584758] LustreError: 95593:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2092.259668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.415213] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2092.426810] LustreError: 93720:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2092.442192] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2092.450406] LustreError: 96055:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2092.455702] LustreError: 96055:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2092.459560] LustreError: 96055:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2092.507575] Lustre: server umount lustre-OST0000 complete [ 2092.509646] LustreError: 96055:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2094.805270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2100.306043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.421730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2100.424559] Lustre: Found index 0 for lustre-OST0000, updating log [ 2100.430485] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2100.488683] Lustre: lustre-OST0000: new disk, initializing [ 2100.495708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2100.730891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.305393] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2106.742287] Lustre: Failing over lustre-OST0000 [ 2106.783417] LustreError: 97143:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2106.787234] LustreError: 97143:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2106.875537] Lustre: server umount lustre-OST0000 complete [ 2110.690480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.698901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.701860] Lustre: Skipped 2 previous similar messages [ 2116.066657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.070860] Lustre: Skipped 1 previous similar message [ 2119.140797] LustreError: 97344:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2119.157377] LustreError: 97344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2119.227941] Lustre: server umount lustre-MDT0000 complete [ 2121.187298] LustreError: 93728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.205245] LustreError: 93728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2122.398992] LustreError: 93713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082188 with bad export cookie 16633879665319266949 [ 2122.404674] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.409815] LustreError: 93713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.662413] Lustre: server umount lustre-MDT0001 complete [ 2130.348663] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:03:15 (1763082195) [ 2141.357317] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2148.572948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.875211] LustreError: 98912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.972532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.558594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2158.212133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2161.985813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2164.387260] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2169.685852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2170.238598] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2174.853989] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2175.346950] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2175.368889] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2175.369087] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2175.380797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2181.675521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2181.722990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2181.773415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2181.825331] Lustre: lustre-OST0001: new disk, initializing [ 2181.829936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2182.076405] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.082980] Lustre: Skipped 2 previous similar messages [ 2186.418712] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2189.360275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2189.365432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2189.447839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2192.952319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2198.899850] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2209.114492] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 2210.283484] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.311967] Lustre: Skipped 3 previous similar messages [ 2214.140638] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 2214.154038] Lustre: Skipped 3 previous similar messages [ 2214.595592] LustreError: 102157:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2214.607472] LustreError: 102157:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2214.870766] Lustre: server umount lustre-MDT0001 complete [ 2215.406294] LustreError: 102070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.444164] LustreError: 102070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2219.052822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2228.247388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.446572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2228.452454] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2228.504719] Lustre: 98899:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2230.061897] LustreError: 102743:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2230.066886] LustreError: 102743:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2230.073156] LustreError: 98904:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2230.181262] LustreError: 102763:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -17 [ 2230.193827] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2230.193827] [ 2232.960276] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2234.343930] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2241.782798] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:05:06 (1763082306) [ 2255.772443] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2272.420276] Lustre: 104769:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2292.822272] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2296.146657] Lustre: 105903:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2387.426595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.431424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.441217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.457198] LustreError: 106377:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2390.462903] LustreError: 106377:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2390.562709] Lustre: server umount lustre-MDT0000 complete [ 2393.059307] LustreError: 102070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.068164] LustreError: 102070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2393.269069] LustreError: 102676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082459 with bad export cookie 16633879665319267873 [ 2393.275734] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.280089] LustreError: 102676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2393.523094] Lustre: server umount lustre-MDT0001 complete [ 2406.542712] Lustre: server umount lustre-OST0000 complete [ 2420.687242] Lustre: server umount lustre-OST0001 complete [ 2425.666541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2433.948181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.690703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2441.905386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2449.720994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.690401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2468.027603] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2477.670934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.688499] Lustre: lustre-MDT0000: reset Object Index mappings [ 2478.005798] LustreError: 109859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.085936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.090153] Lustre: Skipped 1 previous similar message [ 2480.750916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2486.581626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2486.591591] Lustre: lustre-MDT0001: reset Object Index mappings [ 2489.744173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2492.066756] Lustre: 110972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2497.986939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2502.826781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2503.163308] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2510.710367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2512.444854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2512.459522] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2512.470014] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2516.647709] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2523.217638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2526.367420] Lustre: 112831:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2574.192523] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:10:38 (1763082638) [ 2599.395928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.399611] Lustre: Skipped 3 previous similar messages [ 2599.401159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2599.402652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2599.406045] Lustre: Skipped 3 previous similar messages [ 2605.647516] LustreError: 113183:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2605.651688] LustreError: 113183:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2605.805050] Lustre: server umount lustre-MDT0000 complete [ 2609.088763] LustreError: 112153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082675 with bad export cookie 16633879665319315011 [ 2609.093531] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.101929] LustreError: 112153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2609.635465] LustreError: 109855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.639799] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.664204] LustreError: 109855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2609.698414] Lustre: Skipped 3 previous similar messages [ 2615.614653] Lustre: server umount lustre-MDT0001 complete [ 2625.209951] Lustre: server umount lustre-OST0000 complete [ 2642.400140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2642.582859] Lustre: server umount lustre-OST0001 complete [ 2648.111517] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2655.515198] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2663.412674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2669.109895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.405360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2679.904355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2685.770538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2685.865835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.079410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2686.113964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2686.227956] Lustre: lustre-MDT0000: new disk, initializing [ 2686.322557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.330599] Lustre: Skipped 3 previous similar messages [ 2686.370943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2689.892898] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2695.243298] LustreError: 116545:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2695.251236] LustreError: 116545:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2695.545732] Lustre: server umount lustre-MDT0000 complete [ 2697.735276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.300259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.379047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.701898] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2703.719821] Lustre: 117065:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2703.738413] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2707.362687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2712.153664] Lustre: server umount lustre-MDT0000 complete [ 2720.343532] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:13:05 (1763082785) [ 2721.769673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2723.335852] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:13:08 (1763082788) [ 2737.969420] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2745.104837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.415173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2755.211784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2755.257311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2755.313861] Lustre: 119083:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2755.356528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2755.359588] Lustre: Skipped 1 previous similar message [ 2755.414248] Lustre: lustre-MDT0001: new disk, initializing [ 2755.522599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2755.530207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2758.568800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2762.712501] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2768.943953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.993114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2769.290519] Lustre: lustre-OST0000: new disk, initializing [ 2769.297541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2770.834291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2770.857833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2770.935829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2774.376924] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2784.327266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.409074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2784.648525] Lustre: lustre-OST0001: new disk, initializing [ 2784.659246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2785.908213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2785.947651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2786.090154] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2790.135679] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2798.420580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2801.508328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3139.889569] Lustre: 127875:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3139.904562] Lustre: 127875:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3139.912875] Lustre: 127875:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3139.920135] Lustre: 127875:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3139.932811] Lustre: 127875:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3139.946417] Lustre: 127875:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3394.157163] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:24:19 (1763083459) [ 3410.914285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.919449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.924793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.940671] Lustre: Skipped 2 previous similar messages [ 3410.948413] Lustre: Skipped 11 previous similar messages [ 3412.850604] LustreError: 128110:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3412.859799] LustreError: 128110:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3413.005386] Lustre: server umount lustre-MDT0000 complete [ 3415.904893] LustreError: 127906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083482 with bad export cookie 16633879665319380783 [ 3415.908831] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3415.913110] LustreError: 127906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3416.033206] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.036743] LustreError: 122276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.038582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3416.038589] Lustre: Skipped 3 previous similar messages [ 3416.049588] Lustre: Skipped 3 previous similar messages [ 3416.068987] LustreError: 122276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3425.251818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3425.256384] Lustre: Skipped 4 previous similar messages [ 3430.368115] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3430.662511] Lustre: server umount lustre-MDT0001 complete [ 3439.525923] Lustre: server umount lustre-OST0000 complete [ 3448.223437] Lustre: server umount lustre-OST0001 complete [ 3452.552605] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3458.991464] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3465.493733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3470.908542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3476.143323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3481.004621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.414822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.472747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.637553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.669858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.735891] Lustre: lustre-MDT0000: new disk, initializing [ 3486.805580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.808135] Lustre: Skipped 5 previous similar messages [ 3486.833984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.041429] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3497.845337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.901669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.966480] Lustre: 130903:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3498.010749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3498.013746] Lustre: Skipped 1 previous similar message [ 3498.065273] Lustre: lustre-MDT0001: new disk, initializing [ 3498.173913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3498.180273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3500.710675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3505.695597] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3508.103509] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3512.390152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3512.432341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3512.615124] Lustre: lustre-OST0000: new disk, initializing [ 3512.619681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3514.284058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3514.289653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3514.365761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.046694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3520.703875] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3526.420457] Lustre: lustre-MDD0000: changelog on [ 3528.124289] Lustre: lustre-MDD0001: changelog on [ 4496.612671] Lustre: lustre-MDD0001: changelog off [ 4497.413598] Lustre: 131803:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4497.765269] Lustre: lustre-MDD0000: changelog off [ 4501.986479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4501.990094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.995261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4501.997894] Lustre: Skipped 2 previous similar messages [ 4502.501613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4502.501799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.617930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4507.621734] Lustre: Skipped 3 previous similar messages [ 4512.739022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.742032] Lustre: Skipped 1 previous similar message [ 4514.272392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4514.294948] LustreError: 140523:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4514.297169] LustreError: 140523:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4514.352678] Lustre: server umount lustre-OST0000 complete [ 4517.345133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.348158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.977538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.980023] Lustre: Skipped 3 previous similar messages [ 4530.144119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4530.246314] Lustre: server umount lustre-MDT0000 complete [ 4531.675271] LustreError: 130895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084597 with bad export cookie 16633879665320193651 [ 4531.678421] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.680081] LustreError: 130895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4531.813609] Lustre: server umount lustre-MDT0001 complete [ 4535.990424] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4537.181892] Key type lgssc unregistered [ 4537.322622] LNet: 141459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4537.325822] LNetError: 141459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4537.333492] LNet: Removed LNI 192.168.203.145@tcp [ 4537.696196] Key type .llcrypt unregistered [ 4537.698436] Key type ._llcrypt unregistered [ 4545.042410] Key type ._llcrypt registered [ 4545.044061] Key type .llcrypt registered [ 4545.083449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4552.056994] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4552.367499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4552.375633] alg: No test for adler32 (adler32-zlib) [ 4553.241794] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 4553.338360] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4554.928135] Key type lgssc registered [ 4555.367909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4559.174214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.869659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4564.456295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4567.157490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4570.151452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4570.179563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.189588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.314308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4571.332697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4571.379689] Lustre: lustre-MDT0000: new disk, initializing [ 4571.427405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.437466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.176562] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4578.342197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.370897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4578.413977] Lustre: 144607:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4578.445582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4578.447617] Lustre: Skipped 1 previous similar message [ 4578.488119] Lustre: lustre-MDT0001: new disk, initializing [ 4578.527205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4578.542666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4578.547412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4580.212677] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4584.006343] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4585.434712] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4588.196365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.220726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4588.318948] Lustre: lustre-OST0000: new disk, initializing [ 4588.320599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4588.351115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.568296] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4590.718656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4590.721987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4590.754654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4594.068487] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4596.428430] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module obdecho/obdecho [ 4597.212041] LustreError: 147233:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4597.216072] LustreError: 147233:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4597.989965] LustreError: 147329:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4604.385170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4604.389223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.395277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.434575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.434592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.437452] Lustre: Skipped 1 previous similar message [ 4606.444514] Lustre: Skipped 1 previous similar message [ 4608.196972] LustreError: 147669:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4608.247549] Lustre: server umount lustre-MDT0000 complete [ 4609.673746] LustreError: 144600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084675 with bad export cookie 16178917578351032311 [ 4609.677375] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.678257] LustreError: 144600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4609.744638] LustreError: 147870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4609.747692] LustreError: 147870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4609.803135] Lustre: server umount lustre-MDT0001 complete [ 4621.427652] LustreError: 148071:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4621.431365] LustreError: 148071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4621.491503] Lustre: server umount lustre-OST0000 complete [ 4629.955873] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4633.688924] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4637.324475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.646498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.952822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4644.370785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4647.017672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4647.041179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.133080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4647.148325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4647.189418] Lustre: lustre-MDT0000: new disk, initializing [ 4647.221368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.230745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4648.754019] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4653.367839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.389270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.425629] Lustre: 151014:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.453866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4653.456073] Lustre: Skipped 1 previous similar message [ 4653.496312] Lustre: lustre-MDT0001: new disk, initializing [ 4653.538123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.553249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.557027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4655.032836] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4658.211694] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4659.553230] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4661.940245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4661.964026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4662.059224] Lustre: lustre-OST0000: new disk, initializing [ 4662.061468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4662.089592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.035584] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4667.282134] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.738838] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4671.489205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4671.493667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4671.525036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4671.910894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4673.466896] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4673.560916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4676.580808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4676.580866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.588235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4676.590033] Lustre: Skipped 2 previous similar messages [ 4680.430065] LustreError: 153974:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4680.432266] LustreError: 153974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4680.475765] Lustre: server umount lustre-OST0000 complete [ 4684.257057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.260033] LustreError: Skipped 1 previous similar message [ 4684.261546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.266214] Lustre: Skipped 1 previous similar message [ 4684.268449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4687.938239] LustreError: 154175:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4687.940180] LustreError: 154175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4687.968710] Lustre: server umount lustre-MDT0000 complete [ 4689.413627] LustreError: 151795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084755 with bad export cookie 16178917578351033270 [ 4689.418696] LustreError: 151795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4689.418865] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4689.546476] Lustre: server umount lustre-MDT0001 complete [ 4690.933450] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:45:56 (1763084756) [ 4691.492754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4692.153616] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:45:57 (1763084757) [ 4695.070813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.322801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4696.981349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4700.624573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4700.641193] LustreError: 155055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.652035] LustreError: 155055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4702.480988] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4704.788159] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4706.427851] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4709.181130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4711.690996] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4713.862263] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4729.592337] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 4729.595314] Lustre: Skipped 2 previous similar messages [ 4730.849614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.854861] Lustre: Skipped 3 previous similar messages [ 4733.252784] LustreError: 157320:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4733.255842] LustreError: 157320:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4733.297741] Lustre: server umount lustre-MDT0000 complete [ 4734.706409] LustreError: 155050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.790474] LustreError: 155035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084801 with bad export cookie 16178917578351034138 [ 4734.793184] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.795087] LustreError: 155035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4735.968949] LustreError: 155050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.821970] LustreError: 155050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.833316] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 4739.836669] Lustre: Skipped 4 previous similar messages [ 4744.940853] LustreError: 155051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.948629] LustreError: 155051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4749.280175] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.323164] LustreError: 157524:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4749.325184] LustreError: 157524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4749.384562] Lustre: server umount lustre-MDT0001 complete [ 4752.250264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4757.538476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4757.648509] LustreError: 158586:0:(llog.c:1616:llog_backup()) MGC192.168.203.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4757.652364] Lustre: 158586:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4770.477983] LustreError: 158594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.485260] LustreError: 158594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4786.924582] LustreError: 158592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.931401] LustreError: 158592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4818.913382] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4818.916475] Lustre: 158586:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4820.717494] LustreError: 158612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.724139] LustreError: 158612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 4846.048265] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4846.050607] Lustre: 155046:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4859.881171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.884585] Lustre: Skipped 2 previous similar messages [ 4861.296954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4864.480494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.995989] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4866.528257] LustreError: 155046:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4880.359113] LustreError: 142823:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000196497a4 x1848728217556352/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4881.870092] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4883.885832] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4884.475492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4885.479553] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4885.741661] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4887.374437] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4895.716912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.718280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4895.718484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.718491] Lustre: Skipped 3 previous similar messages [ 4895.721905] Lustre: Skipped 2 previous similar messages [ 4899.391921] LustreError: 160071:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4899.394918] LustreError: 160071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4899.431343] Lustre: server umount lustre-MDT0000 complete [ 4900.789652] LustreError: 157525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084967 with bad export cookie 16178917578351040634 [ 4900.794402] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.794896] LustreError: 157525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4900.833085] LustreError: 158592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.839767] LustreError: 158592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 4900.912801] Lustre: server umount lustre-MDT0001 complete [ 4908.445297] Lustre: server umount lustre-OST0000 complete [ 4916.254992] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4920.054755] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4924.359353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.113836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4929.499155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.968633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.404608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.427161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.514627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4934.528211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4934.569295] Lustre: lustre-MDT0000: new disk, initializing [ 4934.592745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.594929] Lustre: Skipped 1 previous similar message [ 4934.600986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4936.023305] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4940.291606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4940.318064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4940.355523] Lustre: 163414:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4940.458153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4941.969472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4945.299426] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4946.663275] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4949.138669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.163555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4950.917160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4950.967984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4951.456678] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4954.783788] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4956.429334] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4956.522931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4958.073771] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4958.156487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4961.251548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4961.251781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.254240] LustreError: Skipped 1 previous similar message [ 4961.254844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.258807] Lustre: Skipped 3 previous similar messages [ 4961.263579] Lustre: Skipped 4 previous similar messages [ 4964.910497] LustreError: 166352:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4964.912280] LustreError: 166352:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4964.945988] Lustre: server umount lustre-OST0000 complete [ 4972.441572] Lustre: server umount lustre-MDT0000 complete [ 4973.733035] LustreError: 164194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085040 with bad export cookie 16178917578351057427 [ 4973.736815] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.737395] LustreError: 164194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4973.851170] Lustre: server umount lustre-MDT0001 complete [ 4976.777249] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:50:42 (1763085042) [ 4979.175638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.674909] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4983.732067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4985.199792] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4986.946478] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4988.173576] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4990.575267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4992.639058] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4994.567180] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5009.208021] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:51:15 (1763085075) [ 5010.402736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.403159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5010.408279] Lustre: Skipped 3 previous similar messages [ 5010.411540] LustreError: Skipped 1 previous similar message [ 5016.402183] Lustre: server umount lustre-OST0000 complete [ 5023.902070] Lustre: server umount lustre-MDT0000 complete [ 5025.187060] LustreError: 167407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085091 with bad export cookie 16178917578351058295 [ 5025.190374] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5025.192026] LustreError: 167407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5025.307996] Lustre: server umount lustre-MDT0001 complete [ 5029.365210] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5030.497122] Key type lgssc unregistered [ 5030.632631] LNet: 170824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5030.635780] LNetError: 170824:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5030.649366] LNet: Removed LNI 192.168.203.145@tcp [ 5030.964647] Key type .llcrypt unregistered [ 5030.966429] Key type ._llcrypt unregistered [ 5038.877312] Key type ._llcrypt registered [ 5038.878919] Key type .llcrypt registered [ 5038.924856] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5039.271757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.287022] alg: No test for adler32 (adler32-zlib) [ 5040.148389] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5040.238703] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5041.824171] Key type lgssc registered [ 5042.207607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5044.995118] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5045.002827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5060.448266] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5065.568167] LustreError: 172024:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 5070.688316] LustreError: 172024:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5070.691440] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5070.692875] LustreError: MGC192.168.203.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5070.695410] LustreError: 172024:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5070.697359] LustreError: 172024:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 5070.699142] LustreError: 172024:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5070.701021] LustreError: 172024:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5070.721387] Lustre: server umount lustre-OST0000 complete [ 5070.723458] LustreError: 172024:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5073.601196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.748361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.036813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5078.904131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5079.026635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.353963] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5082.101356] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5083.324716] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5085.622618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5085.686783] LustreError: 174116:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5085.692168] Lustre: 174116:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.203.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5085.740193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.732274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5089.587626] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.858791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.860214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.862772] Lustre: Skipped 2 previous similar messages [ 5093.867647] Lustre: Skipped 2 previous similar messages [ 5096.445052] LustreError: 174769:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5096.448145] LustreError: 174769:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5096.473127] Lustre: server umount lustre-MDT0000 complete [ 5097.816463] LustreError: 172505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085164 with bad export cookie 11983655046916570488 [ 5097.820893] LustreError: 172505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5097.821129] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.885778] LustreError: 174971:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5097.888501] LustreError: 174971:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5097.939288] Lustre: server umount lustre-MDT0001 complete [ 5109.555661] LustreError: 175172:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5109.558289] LustreError: 175172:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5109.607255] Lustre: server umount lustre-OST0000 complete [ 5112.579350] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5115.947285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5119.124882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5121.383653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.432884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.576106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5127.879118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5127.897994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.980537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5127.994447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5128.030602] Lustre: lustre-MDT0000: new disk, initializing [ 5128.054491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.061047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5129.412306] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5133.604234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5133.628284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5133.662778] Lustre: 177504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5133.688338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5133.690332] Lustre: Skipped 1 previous similar message [ 5133.730438] Lustre: lustre-MDT0001: new disk, initializing [ 5133.766072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5133.779640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5133.784120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5135.177448] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5138.448288] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5139.718810] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5142.233541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5142.263965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5142.365938] Lustre: lustre-OST0000: new disk, initializing [ 5142.368040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5142.397904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.444169] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5147.644839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5147.650481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5147.682431] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5147.686734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5149.170258] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5149.271070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5150.725891] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5150.806382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5152.739937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.740377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5152.740910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5152.743614] Lustre: Skipped 1 previous similar message [ 5153.519112] LustreError: 180441:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5153.520986] LustreError: 180441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5153.560501] Lustre: server umount lustre-OST0000 complete [ 5157.857697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.858501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.862377] Lustre: Skipped 1 previous similar message [ 5157.864537] Lustre: Skipped 1 previous similar message [ 5161.116278] Lustre: server umount lustre-MDT0000 complete [ 5162.408284] LustreError: 178285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085228 with bad export cookie 11983655046916571188 [ 5162.411732] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.412851] LustreError: 178285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5162.481856] LustreError: 180844:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5162.483822] LustreError: 180844:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5162.539301] Lustre: server umount lustre-MDT0001 complete [ 5165.393183] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:53:51 (1763085231) [ 5171.006674] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5172.004543] Key type lgssc unregistered [ 5172.126296] LNet: 182027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5172.128754] LNetError: 182027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5172.136302] LNet: Removed LNI 192.168.203.145@tcp [ 5172.401620] Key type .llcrypt unregistered [ 5172.402860] Key type ._llcrypt unregistered [ 5179.396454] Key type ._llcrypt registered [ 5179.397792] Key type .llcrypt registered [ 5179.435302] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5179.757258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5179.764962] alg: No test for adler32 (adler32-zlib) [ 5180.620803] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5180.708948] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5182.288150] Key type lgssc registered [ 5182.650911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5185.379338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5185.387420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.551594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.876793] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5190.921554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5191.055302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5192.531988] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5194.400442] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5195.620513] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5198.111240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5198.171956] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5198.173502] LustreError: 184853:0:(llog.c:1657:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5198.176425] Lustre: 184853:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.203.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5198.213392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5200.087878] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5201.862397] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5206.500447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.500465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5206.502382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5206.505302] Lustre: Skipped 1 previous similar message [ 5208.702996] LustreError: 185505:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5208.729847] Lustre: server umount lustre-MDT0000 complete [ 5209.969094] LustreError: 183196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085276 with bad export cookie 3460599289424771964 [ 5209.972175] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.973494] LustreError: 183196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5210.040746] LustreError: 185708:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5210.044297] LustreError: 185708:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5210.088530] Lustre: server umount lustre-MDT0001 complete [ 5221.618517] LustreError: 185910:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5221.621187] LustreError: 185910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5221.668992] Lustre: server umount lustre-OST0000 complete [ 5224.830469] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5227.920118] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5231.886279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5234.067830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.178578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5238.310655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5240.681350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5240.703040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.787699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5240.800380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5240.834108] Lustre: lustre-MDT0000: new disk, initializing [ 5240.860711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.868137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5242.255955] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5246.455757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5246.478514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5246.509417] Lustre: 188242:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5246.526833] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5246.528729] Lustre: Skipped 1 previous similar message [ 5246.529774] LustreError: 189008:0:(llog.c:1657:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5246.532048] LustreError: 189008:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5246.533816] Lustre: 189008:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5246.540924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5246.542905] Lustre: Skipped 1 previous similar message [ 5246.574902] Lustre: lustre-MDT0001: new disk, initializing [ 5246.603572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5246.613703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5246.617131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5247.847387] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5250.829700] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.910611] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5254.089035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.116456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.220621] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5254.221724] Lustre: Skipped 1 previous similar message [ 5254.222850] LustreError: 189964:0:(llog.c:1657:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5254.226222] LustreError: 189964:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5254.228074] Lustre: 189964:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5254.231361] Lustre: 189964:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5254.240329] Lustre: lustre-OST0000: new disk, initializing [ 5254.242038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5254.962819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5254.965814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5254.995193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5256.023860] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5258.990090] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5260.274790] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5260.344822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5261.586669] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5261.655442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5265.378859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.379800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5265.383443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.386478] LustreError: Skipped 1 previous similar message [ 5265.388601] Lustre: Skipped 2 previous similar messages [ 5268.333090] LustreError: 191181:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5268.335072] LustreError: 191181:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5268.372188] Lustre: server umount lustre-OST0000 complete [ 5270.498276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.498932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.503266] Lustre: Skipped 2 previous similar messages [ 5270.507609] Lustre: Skipped 2 previous similar messages [ 5275.616855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.618581] Lustre: Skipped 1 previous similar message [ 5275.778045] LustreError: 191382:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5275.780553] LustreError: 191382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5275.807888] Lustre: server umount lustre-MDT0000 complete [ 5277.040206] LustreError: 191384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085343 with bad export cookie 3460599289424772664 [ 5277.043461] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5277.044988] LustreError: 191384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5277.151951] Lustre: server umount lustre-MDT0001 complete [ 5279.911720] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:55:45 (1763085345) [ 5285.103264] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5288.422497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.555479] LustreError: 192950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.560099] LustreError: 192950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5288.570696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.572531] Lustre: Skipped 1 previous similar message [ 5289.870751] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5292.711308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5294.229180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5295.386452] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5297.766154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5300.212207] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5303.143623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5303.165603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5303.221711] Lustre: lustre-OST0001: new disk, initializing [ 5303.224274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5305.170524] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5310.662041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5311.492921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5311.496419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5311.530981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5317.292905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5322.983343] Lustre: DEBUG MARKER: ADD OST3 [ 5324.796321] loop: module loaded [ 5324.816231] loop0: detected capacity change from 0 to 204800000 [ 5324.898354] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5324.903076] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5324.908909] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5324.969787] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5327.072807] loop0: detected capacity change from 0 to 204800000 [ 5328.993103] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5329.015281] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5329.084951] Lustre: lustre-OST0003: new disk, initializing [ 5329.086749] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5329.132757] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5329.137117] Lustre: Skipped 3 previous similar messages [ 5329.878803] Lustre: DEBUG MARKER: STOP OST3 [ 5330.586192] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5330.623266] Lustre: Failing over lustre-OST0003 [ 5330.626598] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -19 [ 5330.630513] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.636067] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 5331.246415] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5332.640096] LustreError: 197281:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5332.652209] LustreError: 197286:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5332.654009] LustreError: 197286:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5332.670446] Lustre: server umount lustre-OST0003 complete [ 5334.062139] LustreError: 194405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.808219] Lustre: DEBUG MARKER: START OST3 again [ 5336.036612] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.038521] LustreError: 194404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5336.046347] LustreError: 194404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5338.271633] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5338.365925] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5339.950660] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5339.962182] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5339.962276] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5339.973285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5339.978420] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5340.420037] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5345.255250] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5354.048738] Lustre: Failing over lustre-OST0003 [ 5354.071489] LustreError: 199087:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5354.074733] LustreError: 199087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5354.099237] Lustre: server umount lustre-OST0003 complete [ 5355.489393] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5355.489818] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.492741] LustreError: Skipped 2 previous similar messages [ 5355.498369] LustreError: 195714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5355.504202] LustreError: 195714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5364.199545] LustreError: 194405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.208322] LustreError: 194405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5369.317048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5369.319964] Lustre: Skipped 4 previous similar messages [ 5373.024750] Lustre: server umount lustre-MDT0000 complete [ 5374.294979] LustreError: 196025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085440 with bad export cookie 3460599289424773532 [ 5374.299469] LustreError: 196025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5374.299654] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.433379] LustreError: 193660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.435294] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.440030] LustreError: 193660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5374.444377] Lustre: Skipped 6 previous similar messages [ 5380.537716] Lustre: server umount lustre-MDT0001 complete [ 5381.937290] Lustre: server umount lustre-OST0000 complete [ 5383.226674] Lustre: server umount lustre-OST0001 complete [ 5385.383780] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5388.176933] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5391.742664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.095657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5396.359161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5398.590677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.986361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5401.009177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.119181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5401.132461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5401.171170] Lustre: lustre-MDT0000: new disk, initializing [ 5401.198566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.201236] Lustre: Skipped 1 previous similar message [ 5401.207844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5402.546288] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5406.929142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.954898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5406.996333] Lustre: 202143:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5407.122524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5408.514928] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5411.690202] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.028886] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.471097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.491738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.761810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5416.802626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5417.578103] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5420.734088] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5422.192495] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5422.279790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5423.781237] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5423.870788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5427.169736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.171022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.172414] LustreError: Skipped 1 previous similar message [ 5427.176814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.180499] Lustre: Skipped 5 previous similar messages [ 5430.639086] LustreError: 205078:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5430.641592] LustreError: 205078:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 5430.675025] Lustre: server umount lustre-OST0000 complete [ 5446.112119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5446.175996] Lustre: server umount lustre-MDT0000 complete [ 5447.505423] LustreError: 202135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085513 with bad export cookie 3460599289424775807 [ 5447.509376] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.509660] LustreError: 202135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5447.641088] Lustre: server umount lustre-MDT0001 complete [ 5448.843574] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:58:34 (1763085514) [ 5454.367286] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5457.506174] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5460.558539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5462.614817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5464.633924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.775367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5469.094945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5469.118209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.195349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5469.205835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5469.208509] Lustre: Skipped 3 previous similar messages [ 5469.242956] Lustre: lustre-MDT0000: new disk, initializing [ 5469.244334] Lustre: Skipped 2 previous similar messages [ 5469.268405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.270651] Lustre: Skipped 2 previous similar messages [ 5469.276757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5469.279397] Lustre: Skipped 2 previous similar messages [ 5470.541630] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5474.490451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5474.514422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5474.545473] Lustre: 208323:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5475.781215] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5478.956375] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5480.308443] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5482.639032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5482.665272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5484.027229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5484.060847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5484.752550] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5487.895653] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5489.310766] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5489.391277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5490.780451] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5490.859271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5494.242469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5494.242888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.245402] LustreError: Skipped 1 previous similar message [ 5494.250217] Lustre: Skipped 3 previous similar messages [ 5494.250893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5494.256111] Lustre: Skipped 7 previous similar messages [ 5497.584105] LustreError: 211262:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5497.586617] LustreError: 211262:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5497.626911] Lustre: server umount lustre-OST0000 complete [ 5505.115081] Lustre: server umount lustre-MDT0000 complete [ 5506.381239] LustreError: 208315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085572 with bad export cookie 3460599289424776689 [ 5506.384736] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.385737] LustreError: 208315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5506.503614] Lustre: server umount lustre-MDT0001 complete [ 5509.294792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.451494] LustreError: 212140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.457448] LustreError: 212140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5510.785400] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5513.703750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5515.164652] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5517.039800] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5518.316551] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5520.695268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5522.873739] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5524.754683] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5544.421073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.423901] LustreError: Skipped 1 previous similar message [ 5547.726786] Lustre: server umount lustre-OST0000 complete [ 5559.776954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5559.779179] Lustre: Skipped 11 previous similar messages [ 5563.360097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5563.419158] Lustre: server umount lustre-MDT0000 complete [ 5563.874427] LustreError: 212158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5563.880480] LustreError: 212158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5564.717269] LustreError: 214405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085631 with bad export cookie 3460599289424777571 [ 5564.719741] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5564.722953] LustreError: 214405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5564.835501] Lustre: server umount lustre-MDT0001 complete [ 5658.752529] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5659.773920] Key type lgssc unregistered [ 5659.900364] LNet: 215482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5659.904066] LNetError: 215482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5659.913359] LNet: Removed LNI 192.168.203.145@tcp [ 5660.219104] Key type .llcrypt unregistered [ 5660.220157] Key type ._llcrypt unregistered [ 5668.563878] Key type ._llcrypt registered [ 5668.564831] Key type .llcrypt registered [ 5668.602837] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5675.136437] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5675.530670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5675.566828] alg: No test for adler32 (adler32-zlib) [ 5676.424782] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5676.530095] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5678.120245] Key type lgssc registered [ 5678.531364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.537397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5683.791525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.898029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.105813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5690.537738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5690.557861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5690.564928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.650310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5691.664511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5691.701391] Lustre: lustre-MDT0000: new disk, initializing [ 5691.728305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.735501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5693.152916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5697.435823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5697.458860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.488670] Lustre: 218783:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5697.513806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5697.515650] Lustre: Skipped 1 previous similar message [ 5697.552283] Lustre: lustre-MDT0001: new disk, initializing [ 5697.585587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.596951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5697.599880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5698.903508] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5702.047711] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.312572] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5705.655822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5705.681823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5705.786161] Lustre: lustre-OST0000: new disk, initializing [ 5705.788845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5705.815840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.694771] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5710.868775] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5712.298693] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5712.378487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5712.381566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5712.418493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5713.397632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5714.842880] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5714.935467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5717.474451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.475339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.476923] LustreError: Skipped 1 previous similar message [ 5717.481291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5717.483187] Lustre: Skipped 1 previous similar message [ 5721.648224] LustreError: 221730:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5721.691743] Lustre: server umount lustre-OST0000 complete [ 5723.104818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5723.107196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.111280] Lustre: Skipped 1 previous similar message [ 5723.113280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.713823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.714782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5729.090314] LustreError: 221932:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5729.092652] LustreError: 221932:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5729.127677] Lustre: server umount lustre-MDT0000 complete [ 5730.381296] LustreError: 219564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085796 with bad export cookie 2226002016375787967 [ 5730.385200] LustreError: 219564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5730.385268] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5730.458798] LustreError: 222134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5730.461146] LustreError: 222134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5730.508301] Lustre: server umount lustre-MDT0001 complete [ 5736.997176] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5740.771382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.936536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.366514] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5745.344430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5745.456192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5746.838831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5747.881229] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5750.229643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5752.410842] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5755.525366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5755.550805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5755.615214] Lustre: lustre-OST0001: new disk, initializing [ 5755.618983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5755.652942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.656139] Lustre: Skipped 1 previous similar message [ 5757.714820] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5763.309076] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5765.453769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5765.457081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5765.491565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5770.147719] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5779.382715] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:04:05 (1763085845) [ 5780.963173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.967546] Lustre: Skipped 2 previous similar messages [ 5780.970356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.973294] Lustre: Skipped 4 previous similar messages [ 5786.081623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.246059] LustreError: 226681:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5786.249215] LustreError: 226681:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5786.282700] Lustre: server umount lustre-MDT0000 complete [ 5787.537066] LustreError: 224782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085853 with bad export cookie 2226002016375788835 [ 5787.540758] LustreError: 224782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5787.541056] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5787.659106] Lustre: server umount lustre-MDT0001 complete [ 5799.411777] LustreError: 227084:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5799.414671] LustreError: 227084:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5799.431132] Lustre: server umount lustre-OST0000 complete [ 5810.996250] LustreError: 227286:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5810.998493] LustreError: 227286:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5811.041766] Lustre: server umount lustre-OST0001 complete [ 5813.169930] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5816.730329] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5820.170482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.544053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5824.730523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5827.002482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.351692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.370939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.465653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5829.480859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5829.521651] Lustre: lustre-MDT0000: new disk, initializing [ 5829.548969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.556119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5831.014710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5835.501439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.521080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5835.555078] Lustre: 229470:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5835.674384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5837.206604] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5840.373188] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5841.712398] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.246720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.269243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5844.380470] Lustre: lustre-OST0000: new disk, initializing [ 5844.382036] Lustre: Skipped 1 previous similar message [ 5844.384449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5844.386338] Lustre: Skipped 2 previous similar messages [ 5845.442117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5845.445670] Lustre: Skipped 1 previous similar message [ 5845.448146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5845.492363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5846.527890] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5849.782787] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.208804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5851.305908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5852.737700] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5852.815873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5855.715150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.715158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.717129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.717493] LustreError: Skipped 1 previous similar message [ 5855.721279] Lustre: Skipped 4 previous similar messages [ 5859.568118] LustreError: 232408:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5859.571337] LustreError: 232408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5859.614640] Lustre: server umount lustre-OST0000 complete [ 5861.344867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5865.953958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.957455] Lustre: Skipped 2 previous similar messages [ 5875.168121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5875.238680] Lustre: server umount lustre-MDT0000 complete [ 5876.193057] LustreError: 229479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.198132] LustreError: 229479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5876.489401] LustreError: 229462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085942 with bad export cookie 2226002016375790865 [ 5876.492594] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5876.493329] LustreError: 229462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5876.613093] Lustre: server umount lustre-MDT0001 complete [ 5879.488871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.650878] LustreError: 233287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.675762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.678698] Lustre: Skipped 2 previous similar messages [ 5881.043172] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5883.991984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5885.517212] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5887.333427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5888.598900] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5890.890968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5893.055249] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5895.078446] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6037.473596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.475573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.476675] LustreError: Skipped 1 previous similar message [ 6037.483795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.486738] Lustre: Skipped 6 previous similar messages [ 6051.296116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6051.313107] LustreError: 235572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6051.316241] LustreError: 235572:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6051.356756] Lustre: server umount lustre-OST0000 complete [ 6058.714749] Lustre: server umount lustre-MDT0000 complete [ 6059.926150] LustreError: 233268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086126 with bad export cookie 2226002016375791747 [ 6059.929246] LustreError: 233268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6059.930480] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6060.053083] Lustre: server umount lustre-MDT0001 complete [ 6159.021280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6160.038819] Key type lgssc unregistered [ 6160.161318] LNet: 236541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6160.164097] LNetError: 236541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6160.171303] LNet: Removed LNI 192.168.203.145@tcp [ 6160.487150] Key type .llcrypt unregistered [ 6160.488140] Key type ._llcrypt unregistered [ 6168.841131] Key type ._llcrypt registered [ 6168.842484] Key type .llcrypt registered [ 6168.878528] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6175.224721] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6175.630481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6175.684070] alg: No test for adler32 (adler32-zlib) [ 6176.549394] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6176.641713] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6178.232206] Key type lgssc registered [ 6178.611287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6181.501250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6183.808037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6185.926970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6188.025443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6193.610711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6197.263993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6197.287137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6197.294296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.406070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6198.421230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6198.462373] Lustre: lustre-MDT0000: new disk, initializing [ 6198.498633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.509306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6199.847977] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6204.061868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6204.091121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6204.128272] Lustre: 240540:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6204.151955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6204.154592] Lustre: Skipped 1 previous similar message [ 6204.200194] Lustre: lustre-MDT0001: new disk, initializing [ 6204.236540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.250074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6204.253876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6205.595617] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6207.891767] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6210.171805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6210.195242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6210.286615] Lustre: lustre-OST0000: new disk, initializing [ 6210.288897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6210.324940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.223498] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6213.239901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6213.242753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6213.290862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6216.392836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6216.420063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6216.494179] Lustre: lustre-OST0001: new disk, initializing [ 6216.496975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6216.538607] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6218.489141] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6221.312800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6221.317595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6221.344809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6223.986589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6227.710738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6236.644414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.645796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6236.652130] Lustre: Skipped 1 previous similar message [ 6241.608307] LustreError: 244061:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6241.643265] Lustre: server umount lustre-MDT0000 complete [ 6242.839394] LustreError: 241320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086309 with bad export cookie 912987622545381886 [ 6242.842050] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.842209] LustreError: 241320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6242.911949] LustreError: 244263:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6242.913662] LustreError: 244263:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6242.962770] Lustre: server umount lustre-MDT0001 complete [ 6254.069082] LustreError: 244463:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6254.070787] LustreError: 244463:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6254.087699] Lustre: server umount lustre-OST0000 complete [ 6265.524466] LustreError: 244665:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6265.526371] LustreError: 244665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6265.569989] Lustre: server umount lustre-OST0001 complete [ 6270.188410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6272.954418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6273.282692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6275.948366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6278.659732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.008699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6285.472387] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6288.905229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.912569] Lustre: lustre-MDT0000: reset Object Index mappings [ 6289.078143] LustreError: 247972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.084265] LustreError: 247972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6289.098293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.444750] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6293.327358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6293.334935] Lustre: lustre-MDT0001: reset Object Index mappings [ 6294.772242] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6295.775492] Lustre: 249082:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6298.031601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6298.175982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.177948] Lustre: Skipped 1 previous similar message [ 6299.237024] LustreError: 249437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.240218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6300.127202] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6303.062579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6304.172106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6305.108354] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6308.305677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6309.507911] Lustre: 250928:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6319.072825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6319.074791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.077682] Lustre: Skipped 2 previous similar messages [ 6319.079196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.080697] Lustre: Skipped 3 previous similar messages [ 6321.854429] LustreError: 251033:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6321.856995] LustreError: 251033:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6321.889088] Lustre: server umount lustre-MDT0000 complete [ 6323.127380] LustreError: 247951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086389 with bad export cookie 912987622545388606 [ 6323.132960] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6323.251790] Lustre: server umount lustre-MDT0001 complete [ 6333.878075] LustreError: 251435:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6333.879898] LustreError: 251435:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6333.896592] Lustre: server umount lustre-OST0000 complete [ 6345.373586] Lustre: server umount lustre-OST0001 complete [ 6351.500837] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:13:37 (1763086417) [ 6356.650198] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6359.851462] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6363.324575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.496034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6367.487841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.650852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.848271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.870563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.949298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6371.959391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6371.997232] Lustre: lustre-MDT0000: new disk, initializing [ 6372.022271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.024087] Lustre: Skipped 1 previous similar message [ 6372.029424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6373.245032] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6377.262903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.286090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6377.318045] Lustre: 255154:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6377.411892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6378.706959] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6381.801134] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6382.993899] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6385.218534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.238864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6385.342480] Lustre: lustre-OST0000: new disk, initializing [ 6385.344699] Lustre: Skipped 1 previous similar message [ 6385.347569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6385.350071] Lustre: Skipped 2 previous similar messages [ 6386.821419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6386.825419] Lustre: Skipped 1 previous similar message [ 6386.828295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6386.869670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6387.287934] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6390.302107] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6391.577694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6391.650636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6392.883919] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6392.949705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6397.410180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6397.411652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.412599] LustreError: Skipped 1 previous similar message [ 6397.420285] Lustre: Skipped 3 previous similar messages [ 6397.421102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6397.423949] Lustre: Skipped 4 previous similar messages [ 6399.663347] LustreError: 258090:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6399.665603] LustreError: 258090:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6399.699065] Lustre: server umount lustre-OST0000 complete [ 6402.529920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.530661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.533052] Lustre: Skipped 2 previous similar messages [ 6402.537682] Lustre: Skipped 1 previous similar message [ 6407.649313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6407.651767] Lustre: Skipped 1 previous similar message [ 6415.328081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6415.388539] Lustre: server umount lustre-MDT0000 complete [ 6416.599650] LustreError: 255931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086482 with bad export cookie 912987622545390699 [ 6416.603060] LustreError: 255931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6416.604701] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.743749] Lustre: server umount lustre-MDT0001 complete [ 6422.602145] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6425.980814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.111708] LustreError: 259666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.125761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.127492] Lustre: Skipped 2 previous similar messages [ 6427.309894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6430.126723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6431.563931] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6432.532818] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6434.652123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6436.496337] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6439.191046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.219582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6439.275454] Lustre: lustre-OST0001: new disk, initializing [ 6439.277066] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6440.962204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6440.965661] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6440.998463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6441.060416] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6445.350780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6446.426766] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6453.123862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6453.128561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6455.777243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6455.780227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.787631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6455.789487] Lustre: Skipped 2 previous similar messages [ 6460.868370] LustreError: 263169:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6460.870986] LustreError: 263169:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6460.899678] Lustre: server umount lustre-MDT0000 complete [ 6461.409969] LustreError: 259666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.416193] LustreError: 259666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6462.072403] LustreError: 259648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086528 with bad export cookie 912987622545391581 [ 6462.075974] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6462.076495] LustreError: 259648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6462.216474] Lustre: server umount lustre-MDT0001 complete [ 6473.730856] Lustre: server umount lustre-OST0000 complete [ 6485.209669] Lustre: server umount lustre-OST0001 complete [ 6490.359805] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6493.691299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.835796] LustreError: 264950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6493.855053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.856829] Lustre: Skipped 3 previous similar messages [ 6495.114049] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6497.935947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6499.346247] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6500.286500] Lustre: 266057:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6502.371585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6504.363180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6504.553426] LustreError: 266792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.560893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6507.123141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6508.283797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6509.140951] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6512.202061] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6513.383371] Lustre: 267910:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6520.205481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6520.210232] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6523.268682] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:16:29 (1763086589) [ 6539.237856] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.240426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6539.240752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.240762] Lustre: Skipped 4 previous similar messages [ 6539.241113] Lustre: Skipped 5 previous similar messages [ 6540.925263] LustreError: 268620:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6540.927714] LustreError: 268620:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6540.957659] Lustre: server umount lustre-MDT0000 complete [ 6542.124042] LustreError: 264929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086608 with bad export cookie 912987622545397853 [ 6542.126599] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.127136] LustreError: 264929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6544.353297] LustreError: 266791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.357344] LustreError: 266791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6548.267814] Lustre: server umount lustre-MDT0001 complete [ 6555.587651] Lustre: server umount lustre-OST0000 complete [ 6562.962335] Lustre: server umount lustre-OST0001 complete [ 6565.031201] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6568.318849] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6571.995540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6574.049469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6576.067808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6578.261794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6583.338949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6586.820210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6586.841139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.936579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6586.947497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6586.985552] Lustre: lustre-MDT0000: new disk, initializing [ 6587.014607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6588.279680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6592.556767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.580874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6592.612241] Lustre: 272103:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6592.718183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6593.977748] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6598.323994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.355273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6598.953990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6600.326685] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6604.380172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6604.406086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6606.249067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6606.455774] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6610.923552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6620.996456] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:18:06 (1763086686) [ 6621.667289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.670497] Lustre: Skipped 4 previous similar messages [ 6621.673854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.675433] Lustre: Skipped 10 previous similar messages [ 6627.814672] Lustre: server umount lustre-MDT0000 complete [ 6629.039379] LustreError: 273707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086695 with bad export cookie 912987622545405714 [ 6629.042679] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6629.044910] LustreError: 273707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6629.172725] Lustre: server umount lustre-MDT0001 complete [ 6640.136381] Lustre: server umount lustre-OST0000 complete [ 6651.609607] Lustre: server umount lustre-OST0001 complete [ 6657.269707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6660.244990] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6663.402697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6665.427785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.408048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.389938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.843795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.866465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.944039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6671.946814] Lustre: Skipped 2 previous similar messages [ 6671.954575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6671.956117] Lustre: Skipped 4 previous similar messages [ 6671.989860] Lustre: lustre-MDT0000: new disk, initializing [ 6671.991050] Lustre: Skipped 3 previous similar messages [ 6672.015500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.018616] Lustre: Skipped 7 previous similar messages [ 6672.024659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6672.026961] Lustre: Skipped 3 previous similar messages [ 6673.236339] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6677.129844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.151502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6677.179552] Lustre: 279157:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6677.270851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6677.272978] Lustre: Skipped 2 previous similar messages [ 6678.435406] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6681.323618] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6682.398072] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6684.519478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6684.540758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6686.428246] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6686.686529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.467599] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.134821] Lustre: Mounted lustre-client [ 6690.432251] LustreError: 281660:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88e9c4e48000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6690.439484] LustreError: 281660:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6690.441829] LustreError: 281660:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 6690.461100] Lustre: Unmounted lustre-client [ 6695.394631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.396300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.398540] Lustre: Skipped 3 previous similar messages [ 6695.400065] Lustre: Skipped 4 previous similar messages [ 6699.281431] Lustre: server umount lustre-OST0000 complete [ 6706.663856] Lustre: server umount lustre-MDT0000 complete [ 6707.898295] LustreError: 279148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086774 with bad export cookie 912987622545407240 [ 6707.901525] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.904515] LustreError: 279148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6708.019260] Lustre: server umount lustre-MDT0001 complete [ 6711.367402] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6712.233394] Key type lgssc unregistered [ 6743.072549] LNet: 282936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6743.076941] LNetError: 282936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6820.902355] LNet: Removed LNI 192.168.203.145@tcp [ 6821.202113] Key type .llcrypt unregistered [ 6821.204103] Key type ._llcrypt unregistered [ 6826.848576] Key type ._llcrypt registered [ 6826.849528] Key type .llcrypt registered [ 6826.882621] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6833.130193] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6833.435536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6833.518077] alg: No test for adler32 (adler32-zlib) [ 6834.354564] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6834.434890] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6836.008164] Key type lgssc registered [ 6836.381807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6839.405763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.545389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6843.590141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6845.653894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6846.227354] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:21:52 (1763086912) [ 6848.385586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6848.412662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6848.416929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.501353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6849.514410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6849.555997] Lustre: lustre-MDT0000: new disk, initializing [ 6849.581685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.589272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6850.978342] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6854.979088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.004783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6855.040864] Lustre: 286116:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6855.088559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6855.092079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6855.093581] Lustre: Skipped 1 previous similar message [ 6855.126598] Lustre: lustre-MDT0001: new disk, initializing [ 6855.161371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6855.174582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6855.177705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6856.370525] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6859.325435] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.382245] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6862.557092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6862.587560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6862.700774] Lustre: lustre-OST0000: new disk, initializing [ 6862.703068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6862.737636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6864.378898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6864.382240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6864.429883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6864.584707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6867.497950] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6874.595156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6874.595741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.596558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6874.597284] LustreError: Skipped 1 previous similar message [ 6874.606981] Lustre: Skipped 1 previous similar message [ 6877.422095] LustreError: 288558:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6877.459452] Lustre: server umount lustre-OST0000 complete [ 6879.715121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.715983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.718468] Lustre: Skipped 1 previous similar message [ 6879.720078] Lustre: Skipped 1 previous similar message [ 6884.833258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6884.835645] Lustre: Skipped 1 previous similar message [ 6889.953339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6889.955074] Lustre: Skipped 1 previous similar message [ 6893.024123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6893.060373] LustreError: 288759:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6893.062840] LustreError: 288759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6893.093931] Lustre: server umount lustre-MDT0000 complete [ 6894.301368] LustreError: 286108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086960 with bad export cookie 9777566918683026998 [ 6894.305407] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.305640] LustreError: 286108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6894.372268] LustreError: 288963:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6894.375512] LustreError: 288963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6894.426793] Lustre: server umount lustre-MDT0001 complete [ 6898.076843] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6899.054380] Key type lgssc unregistered [ 6899.167308] LNet: 289495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6899.169308] LNetError: 289495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.175302] LNet: Removed LNI 192.168.203.145@tcp [ 6899.443650] Key type .llcrypt unregistered [ 6899.444598] Key type ._llcrypt unregistered [ 6941.737761] Key type ._llcrypt registered [ 6941.739421] Key type .llcrypt registered [ 6941.805126] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6941.974349] Key type .llcrypt unregistered [ 6941.976636] Key type ._llcrypt unregistered [ 6976.288840] Key type ._llcrypt registered [ 6976.290022] Key type .llcrypt registered [ 6976.333666] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6976.479366] Key type .llcrypt unregistered [ 6976.481334] Key type ._llcrypt unregistered [ 6988.796431] Key type ._llcrypt registered [ 6988.798050] Key type .llcrypt registered [ 6988.859691] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6989.022663] Key type .llcrypt unregistered [ 6989.025079] Key type ._llcrypt unregistered [ 7003.107044] Key type ._llcrypt registered [ 7003.108423] Key type .llcrypt registered [ 7003.148195] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7003.272358] Key type .llcrypt unregistered [ 7003.273811] Key type ._llcrypt unregistered [ 7036.312926] Key type ._llcrypt registered [ 7036.314112] Key type .llcrypt registered [ 7036.353772] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7036.479247] Key type .llcrypt unregistered [ 7036.481043] Key type ._llcrypt unregistered [ 7048.547177] Key type ._llcrypt registered [ 7048.548136] Key type .llcrypt registered [ 7048.579288] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7048.663361] Key type .llcrypt unregistered [ 7048.664810] Key type ._llcrypt unregistered [ 7061.236424] Key type ._llcrypt registered [ 7061.238451] Key type .llcrypt registered [ 7061.310805] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7061.488452] Key type .llcrypt unregistered [ 7061.490745] Key type ._llcrypt unregistered [ 7074.731556] Key type ._llcrypt registered [ 7074.732751] Key type .llcrypt registered [ 7074.776716] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7074.908295] Key type .llcrypt unregistered [ 7074.909981] Key type ._llcrypt unregistered [ 7088.648783] Key type ._llcrypt registered [ 7088.650206] Key type .llcrypt registered [ 7088.704881] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7088.855382] Key type .llcrypt unregistered [ 7088.858080] Key type ._llcrypt unregistered [ 7100.976093] Key type ._llcrypt registered [ 7100.977088] Key type .llcrypt registered [ 7101.015594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7101.112687] Key type .llcrypt unregistered [ 7101.113710] Key type ._llcrypt unregistered [ 7121.019845] Key type ._llcrypt registered [ 7121.020990] Key type .llcrypt registered [ 7121.055996] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7121.143258] Key type .llcrypt unregistered [ 7121.145204] Key type ._llcrypt unregistered [ 7131.940392] Key type ._llcrypt registered [ 7131.941299] Key type .llcrypt registered [ 7131.972236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7132.063187] Key type .llcrypt unregistered [ 7132.064150] Key type ._llcrypt unregistered [ 7142.798508] Key type ._llcrypt registered [ 7142.799885] Key type .llcrypt registered [ 7142.841982] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7142.949170] Key type .llcrypt unregistered [ 7142.950656] Key type ._llcrypt unregistered [ 7163.112413] Key type ._llcrypt registered [ 7163.113666] Key type .llcrypt registered [ 7163.148780] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7169.691909] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7170.103547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7170.116496] alg: No test for adler32 (adler32-zlib) [ 7170.977642] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7171.074769] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7172.656185] Key type lgssc registered [ 7173.140933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7176.069992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7178.748199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7180.620270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.848053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7183.718661] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6959 sec ========== 21:27:29 (1763087249) [ 7184.596246] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:27:30 (1763087250) === [ 7185.611404] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:27:31 (1763087251) === [ 7199.626839] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7200.666316] Key type lgssc unregistered [ 7200.786343] LNet: 311197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7200.788402] LNetError: 311197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7200.799278] LNet: Removed LNI 192.168.203.145@tcp [ 7201.051674] Key type .llcrypt unregistered [ 7201.052628] Key type ._llcrypt unregistered